PaDaS-Lab/webfaq-v2-bitexts

<h1 align="center">WebFAQ 2.0 Bilingual Datasets (Bitexts)</h1>

<h4 align="center"> <p> <a href=#overview>Overview</a> | <a href=#whats-new-in-v20>What's New in v2.0</a> | <a href=#details>Details</a> | <a href=#construction-method>Construction Method</a> | <a href=#structure>Structure</a> | <a href=#examples>Examples</a> | <a href=#considerations>Considerations</a> | <a href=#license>License</a> | <a href=#citation>Citation</a> | <a href=#contact>Contact</a> <p> </h4>

Note: Note that the SIGIR Resource submission reports 104 languages, however, after re-uploading the WebFAQ 2.0 dataset, it now includes 108 languages in total. Furthermore note that for the Bilingual Datasets, we now include all those language combinations with at least 100 samples (instead of 1,000). This increases the number of combinations from 3,118 to 3,970. The total number of samples across all 3,970 combinations is therefore 14.3M (instead of 13.8M, as reported in the submission).

Overview

The WebFAQ 2.0 Bilingual Datasets (Bitexts) are derived from WebFAQ 2.0, a large-scale multilingual FAQ corpus containing 198 million QA pairs across 108 languages.

👉 WebFAQ 2.0 (monolingual QA dataset): https://huggingface.co/datasets/michaeldinzinger/webfaq-v2

This repository contains aligned question–answer pairs across languages, mined using multilingual sentence embeddings and strict similarity thresholds.

Compared to the previous bitext release:

👉 WebFAQ v1 Bitexts (previous version): https://huggingface.co/datasets/PaDaS-Lab/webfaq-bitexts

WebFAQ 2.0 Bitexts dramatically expand coverage in both scale and language diversity.

Why is this dataset useful?

• —🌍 Large-scale cross-lingual QA alignments
• —🔎 Cross-lingual Information Retrieval (CLIR)
• —🧠 Training multilingual sentence embeddings
• —🌐 Bitext mining research
• —🔁 Neural Machine Translation

What's New in v2.0

WebFAQ 2.0 introduces major improvements over the previous bitext release:

1️⃣ Massive Scale Expansion

• —14.3 million aligned QA pairs (vs. 1.5M in v1)
• —3,970 language combinations (≥100 aligned samples each)
• —3,118 language pairs contain ≥1,000 and 1,282 language pairs contain ≥4,000 aligned QA pairs

This represents nearly a 10× increase in alignment size.

2️⃣ Improved Multilingual Coverage

• —108 source languages (vs. 75 in v1)
• —Strong growth in:

• —Non-English ↔ Non-English alignments
• —Underrepresented language pairs
• —Regional language combinations (e.g., Marathi–Telugu)

The largest alignment growth is no longer limited to English-centered pairs.

3️⃣ New Crawling Strategy (Inherited from WebFAQ 2.0)

Instead of relying solely on Web Data Commons dumps, WebFAQ 2.0:

• —Mines FAQPage markup directly from Common Crawl
• —Crawls pages using OWLer (OpenWebSearch project)
• —Extracts:

• —Questions
• —Answers
• —Page titles
• —Page descriptions
• —hreflang links for multilingual alignment

This yields richer contextual metadata and improved cross-lingual coverage .

Details

Language Coverage

Each dataset configuration corresponds to one language pair, e.g.:

• —eng-deu
• —spa-por
• —rus-ukr
• —mar-tel
• —etc.

Statistics:

• —3,970 language pairs with ≥100 aligned QA pairs
• —3,118 language pairs with ≥1,000 aligned QA pairs
• —1,282 language pairs with ≥4,000 aligned QA pairs
• —Total: 14.3M aligned QA pairs

Language pairs are not restricted to English-centered combinations. Many alignments exist between:

• —Romance languages
• —Slavic languages
• —Indo-Aryan languages
• —Southeast Asian languages
• —etc.

<details> <summary>📋 Table of all language pairs (lexicographical order)</summary>