CoolFace
Datasetpublic

hotchpotch/wikipedia-multilingual-ir-pairs

wikipedia-multilingual-ir-pairs This dataset is designed for multilingual information retrieval training. Compared with raw Wikipedia paragraph dumps, it provides cleaner and more practical supervision by pairing title/section-driven queries with relevant paragraph-level documents and applying rule-based filtering to remove low-value sections and noisy fragments. It is intended for IR model training, including contrastive learning and retrieval/reranking objectives.… See the full description on the dataset page: https://huggingface.co/datasets/hotchpotch/wikipedia-multilingual-ir-pairs.

sourceHugging Faceupdated 4mo agoView on Hugging Face
0likes907downloads
Dataset Card

wikipedia-multilingual-ir-pairs

This dataset is designed for multilingual information retrieval training. Compared with raw Wikipedia paragraph dumps, it provides cleaner and more practical supervision by pairing title/section-driven queries with relevant paragraph-level documents and applying rule-based filtering to remove low-value sections and noisy fragments. It is intended for IR model training, including contrastive learning and retrieval/reranking objectives.

Dataset at a glance

  • —Task: IR training (retrieval and reranking)
  • —Fields: query (Wikipedia title, or title + section header), document (paragraph text from the corresponding article section)
  • —Example query construction: if title is Kyoto and section header is History, the query becomes Kyoto History and is paired with the corresponding section text.
  • —Total rows: 95,757,423
  • —Language subsets: 11
  • —Source dataset: singletongue/wikipedia-paragraphs

Language Subsets

SubsetLanguageRows
arwiki-20251202-v1.1.0Arabic2,985,795
dewiki-20251202-v1.1.0German12,740,626
enwiki-20251202-v1.1.0English29,568,452
eswiki-20251202-v1.1.0Spanish8,514,623
frwiki-20251202-v1.1.0French11,282,018
itwiki-20251202-v1.1.0Italian6,941,259
jawiki-20251202-v1.1.0Japanese5,864,346
kowiki-20251202-v1.1.0Korean1,777,129
ptwiki-20251202-v1.1.0Portuguese3,937,511
ruwiki-20251202-v1.1.0Russian8,299,364
zhwiki-20251202-v1.1.0Chinese3,846,300

Dataset Creation Process (Rough)

The pipeline is implemented in `scripts/build_subheader_filtered_hf_ds.py`.

  1. 1.Load each latest language subset from singletongue/wikipedia-paragraphs.
  2. 2.Filter out non-article pages and low-value sections (for example, "See also", references, external links; language-specific rules).
  3. 3.Apply additional quality filters (short/noisy list fragment removal, disambiguation lead gates, query-length constraints, and dropping pairs where len(query) > len(document)).
  4. 4.Split and merge paragraphs into practical chunks (character-length targets: 1000 for non-CJK, 600 for Japanese/Korean, 500 for Chinese; newline-aware splitting with an optional split cap; balanced merge for consecutive paragraphs in the same section; and a final hard cap on document length).
  5. 5.Save each language subset with query / document columns.

Acknowledgements

Thank you for publishing the very useful dataset singletongue/wikipedia-paragraphs. This dataset made it possible to build this IR-focused dataset quickly.

License

This dataset is derived from Wikipedia text and follows Wikipedia licensing:

  • —Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)
  • —GNU Free Documentation License (GFDL), where applicable

Downstream users should preserve attribution and share-alike obligations for the Wikipedia-derived text.