SaylorTwift/mteb-bitext-mining-aggregated

MTEB BitextMining Aggregated Dataset (Full)

This dataset aggregates ALL configs from 10 BitextMining datasets in the MTEB (Massive Text Embedding Benchmark) Multilingual v2 benchmark into a single, unified dataset for comprehensive bitext mining evaluation.

Dataset Summary

• —Total Examples: 448,229 sentence pairs
• —Source Datasets (Configs): 10 MTEB BitextMining tasks
• —Total Splits: 332 language pairs/configurations
• —Languages: 300+ unique language codes across all datasets
• —Task: Bitext Mining (parallel sentence retrieval)
• —Format: Standardized schema across all sources

Structure

Each source dataset is a config, and each original config (language pair) within that dataset is a split.

Example Usage

from datasets import load_dataset

# Load specific config (source dataset)
tatoeba = load_dataset("SaylorTwift/mteb-bitext-mining-aggregated", "Tatoeba")
# This gives you 112 splits, one for each language pair

# Access a specific language pair split
french_english = tatoeba['fra-eng']
print(f"French-English pairs: {len(french_english)}")

# Load another config
indic = load_dataset("SaylorTwift/mteb-bitext-mining-aggregated", "IndicGenBenchFloresBitextMining")
# This gives you 58 splits for different Indic language pairs

# Access a split
hindi_english = indic['hin-eng']

Schema

Each example contains:

• —sentence1 (string): First sentence of the pair
• —sentence2 (string): Second sentence of the pair (translation/parallel text)
• —lang (string): Language pair code (e.g., "fra-eng", "de-en")
• —source_dataset (string): Original MTEB dataset name
• —original_split (string): Original split name (train/validation/test)
• —config (string): Original config name

Configs (Source Datasets)

Total: 10 configs, 332 splits, 448,229 examples

Example Splits by Config

Tatoeba (112 language pairs)

sqi-eng, fry-eng, kur-eng, tur-eng, deu-eng, ell-eng, spa-eng, fra-eng, ita-eng, jpn-eng, cmn-eng, kor-eng, ara-eng, rus-eng, por-eng, hin-eng, etc.

IN22GenBitextMining (128 Indic pairs)

asm_Beng-ben_Beng, asm_Beng-eng_Latn, ben_Beng-hin_Deva, guj_Gujr-mar_Deva, etc. (all combinations of 23 Indic languages)

IndicGenBenchFloresBitextMining (58 pairs)

asm-eng, awa-eng, ben-eng, bgc-eng, bho-eng, bod-eng, guj-eng, hin-eng, kan-eng, mal-eng, mar-eng, nep-eng, ory-eng, pan-eng, tam-eng, tel-eng, urd-eng, etc.

BUCC_v2 (4 language pairs)

de-en, fr-en, ru-en, zh-en

NusaTranslationBitextMining (11 Indonesian languages)

ind-abs, ind-bew, ind-bhp, ind-btk, ind-jav, ind-mad, ind-mak, ind-min, ind-mui, ind-rej, ind-sun

NusaXBitextMining (11 pairs)

eng-ace, eng-ban, eng-bbc, eng-bjn, eng-bug, eng-ind, eng-jav, eng-mad, eng-min, eng-nij, eng-sun

Usage Examples

Load all language pairs from a specific source

from datasets import load_dataset

# Load all Tatoeba language pairs
tatoeba = load_dataset("SaylorTwift/mteb-bitext-mining-aggregated", "Tatoeba")

# Iterate through all language pairs
for lang_pair, dataset in tatoeba.items():
    print(f"{lang_pair}: {len(dataset)} pairs")

Load a specific language pair

# Load just German-English from BUCC
bucc = load_dataset("SaylorTwift/mteb-bitext-mining-aggregated", "BUCC_v2")
de_en = bucc['de-en']

for example in de_en:
    print(f"DE: {example['sentence1']}")
    print(f"EN: {example['sentence2']}")
    print()

Filter by language across all datasets

# Load Tatoeba
tatoeba = load_dataset("SaylorTwift/mteb-bitext-mining-aggregated", "Tatoeba")

# Get all examples for a specific language pair
french_english = tatoeba['fra-eng']
print(f"Found {len(french_english)} French-English pairs")

Excluded Datasets

BibleNLPBitextMining (828 configs, 900+ languages) was excluded due to incompatible schema that uses language codes as column names instead of the standard sentence1/sentence2 format.

FloresBitextMining and NTREXBitextMining were excluded in the previous version but may be revisitable with updated processing.

Citation

If you use this dataset, please cite the MTEB benchmark:

@article{muennighoff2022mteb,
  title={MTEB: Massive Text Embedding Benchmark},
  author={Muennighoff, Niklas and Tazi, Nouamane and Magne, Lo{\"\i}c and Reimers, Nils},
  journal={arXiv preprint arXiv:2210.07316},
  year={2022}
}

Individual Dataset Citations

Tatoeba

@inproceedings{artetxe2019massively,
  title={Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond},
  author={Artetxe, Mikel and Schwenk, Holger},
  booktitle={Transactions of the Association for Computational Linguistics},
  year={2019}
}

BUCC

@inproceedings{zweigenbaum2017overview,
  title={Overview of the second BUCC shared task: Spotting parallel sentences in comparable corpora},
  author={Zweigenbaum, Pierre and Sharoff, Serge and Rapp, Reinhard},
  booktitle={Proceedings of the 10th workshop on building and using comparable corpora},
  year={2017}
}

Additional citations available in the original MTEB task metadata and individual dataset pages.

Dataset Statistics

Language Coverage

• —Total unique language codes: 300+
• —Language families: Indo-European, Sino-Tibetan, Afro-Asiatic, Austronesian, Dravidian, and many more
• —Coverage: High-resource (English, French, German, Spanish, Chinese, etc.), mid-resource (Hindi, Bengali, Tamil, etc.), and low-resource languages

Split Distribution

• —Total splits: 332 (each representing a specific language pair or configuration)
• —Examples per split: Ranges from 228 to 8,750, with most splits containing 500-1,000 examples

Data Quality

• —All sentence pairs have been validated to contain non-empty sentence1 and sentence2 fields
• —Language codes are preserved from original datasets
• —Source attribution maintained for every example

License

This aggregated dataset inherits the licenses from its source datasets. Most MTEB datasets are released under permissive licenses (Apache 2.0, MIT, CC-BY, etc.). Please refer to the original dataset pages for specific licensing information.

Acknowledgments

• —MTEB Team: For creating and maintaining the benchmark
• —Original Dataset Creators: For providing high-quality bitext mining datasets
• —Hugging Face: For dataset hosting and infrastructure

Version History

• —v2.0 (2026-04-02): Full release
• —10 source datasets (configs)
• —332 splits (all language pairs)
• —448,229 sentence pairs
• —300+ language codes

• —v1.0 (2026-04-02): Initial partial release (deprecated)
• —Only loaded default configs
• —8 source datasets
• —139,457 examples

Contact

For questions or issues with this aggregated dataset, please open an issue on the repository or contact the dataset creator.