PaDaS-Lab/webfaq-retrieval

<h1 align="center">WebFAQ Retrieval Dataset</h1> <h4 align="center"> <p> <a href=#overview>Overview</a> | <a href=#details>Details</a> | <a href=#structure>Structure</a> | <a href=#examples>Examples</a> | <a href=#considerations>Considerations</a> | <a href=#license>License</a> | <a href=#citation>Citation</a> | <a href=#contact>Contact</a> | <a href=#acknowledgement>Acknowledgement</a> <p> </h4>

Overview

The WebFAQ Retrieval Dataset is a carefully filtered and curated subset of the broader WebFAQ Q&A Dataset. It is purpose-built for Information Retrieval (IR) tasks, such as training and evaluating dense or sparse retrieval models in multiple languages.

Each of the 49 largest languages from the WebFAQ corpus has been thoroughly cleaned and refined to ensure an unblurred notion of relevance between a query (question) and its corresponding document (answer). In particular, we applied:

• —Deduplication of near-identical questions,
• —Semantic consistency checks for question-answer alignment,
• —Train/Test splits for retrieval experiments.

Details

Languages

The WebFAQ Retrieval Dataset covers 49 languages from the original WebFAQ corpus, both high-resource as well as low-resource. To ensure diversity, the 49 subsets originate from at least 100 different websites each. A single language comprises a few thousands to a few million of QA pairs after our rigorous filtering steps:

Top 20 Languages

<details> <summary> Table of all 49 languages (lexicographical order) </summary>

</details>

<details> <summary> Table of all 49 languages (ordered by size) </summary>

</details>

<details> <summary> NDCG@10 in % for three examplary multilingual embedding models </summary>

</details>

Structure

Unlike the raw Q&A dataset, WebFAQ Retrieval provides explicit train/test splits for each of the 49 languages. The general structure for each language is:

• —Corpus: A set of unique documents (answers) with IDs and text fields.
• —Queries: A set of question strings, each tied to a document ID for relevance.
• —Qrels: Relevance labels, mapping each question to its relevant document (corresponding answer).

Folder Layout (e.g., for eng)

eng/
  ├── corpus.jsonl        # all unique documents (answers)
  ├── queries.jsonl       # all queries for train/test
  ├── train.jsonl         # relevance annotations for train
  └── test.jsonl          # relevance annotations for test

Examples

Below is a small snippet showing how to load English train/test sets with 🤗 Datasets:

import json
from datasets import load_dataset
from tqdm import tqdm

# Load train qrels
train_qrels = load_dataset(
    "PaDaS-Lab/webfaq-retrieval",
    "eng-qrels",
    split="train"
)

# Inspect first qrel
print(json.dumps(train_qrels[0], indent=4))

# Load the corpus (answers)
data_corpus = load_dataset(
    "PaDaS-Lab/webfaq-retrieval",
    "eng-corpus",
    split="corpus"
)
corpus = {
    d["_id"]: {"title": d["title"], "text": d["text"]} for d in tqdm(data_corpus)
}

# Inspect first document
print("Document:")
print(json.dumps(corpus[train_qrels[0]["corpus-id"]], indent=4))

# Load all queries
data_queries = load_dataset(
    "PaDaS-Lab/webfaq-retrieval",
    "eng-queries",
    split="queries"
)
queries = {
    q["_id"]: q["text"] for q in tqdm(data_queries)
}

# Inspect first query
print("Query:")
print(json.dumps(queries[train_qrels[0]["query-id"]], indent=4))

# Keep only those queries with relevance annotations
query_ids = set([q["query-id"] for q in train_qrels])
queries = {
    qid: query for qid, query in queries.items() if qid in query_ids
}
print(f"Number of queries: {len(queries)}")

Below is a code snippet showing how to evaluate retrieval performance using the mteb library:

from mteb import MTEB
from mteb.tasks.Retrieval.multilingual.WebFAQRetrieval import WebFAQRetrieval

# ... Load model ...

# Load the WebFAQ task
task = WebFAQRetrieval()
eval_split = "test"

evaluation = MTEB(tasks=[task])
evaluation.run(
    model,
    eval_splits=[eval_split],
    output_folder="output",
    overwrite_results=True
)

Considerations

Please note the following considerations when using the collected QAs:

• —[Q&A Dataset] Risk of Duplicate or Near-Duplicate Content: The raw Q&A dataset is large and includes minor paraphrases.
• —[Retrieval Dataset] Sparse Relevance: As raw FAQ data, each question typically has one “best” (on-page) answer. Additional valid answers may exist on other websites but are not labeled as relevant.
• —Language Detection Limitations: Some QA pairs mix languages, or contain brand names, which can confuse automatic language classification.
• —No Guarantee of Factual Accuracy: Answers reflect the content of the source websites. They may include outdated, biased, or incorrect information.
• —Copyright and Privacy: Please ensure compliance with any applicable laws and the source website’s terms.

License

The Collection of WebFAQ Datasets is shared under **Creative Commons Attribution 4.0 (CC BY 4.0)** license.

Note: The dataset is derived from public webpages in Common Crawl snapshots (2022–2024) and intended for research purposes. Each FAQ’s text is published by the original website under their terms. Downstream users should verify any usage constraints from the original websites as well as Common Crawl’s Terms of Use.

Citation

If you use this dataset in your research, please consider citing the associated paper:

@misc{dinzinger2025webfaq,
    title={WebFAQ: A Multilingual Collection of Natural Q&A Datasets for Dense Retrieval},
    author={Michael Dinzinger and Laura Caspari and Kanishka Ghosh Dastidar and Jelena Mitrović and Michael Granitzer},
    year={2025},
    eprint={2502.20936},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

Contact

For inquiries and feedback, please feel free to contact us via E-Mail (michael.dinzinger@uni-passau.de) or start a discussion on HuggingFace or GitHub.

Acknowledgement

We thank the Common Crawl and Web Data Commons teams for providing the underlying data, and all contributors who helped shape the WebFAQ project.

Thank you

We hope the Collection of WebFAQ Datasets serves as a valuable resource for your research. Please consider citing it in any publications or projects that use it. If you encounter issues or want to contribute improvements, feel free to get in touch with us on HuggingFace or GitHub.

Happy researching!