AdaMLLab/HinMix
HinMix (https://arxiv.org/abs/2512.18834) is a Hindi pretraining corpus containing 76 billion tokens across 60 million documents (in the minhash subset). Rather than scraping the web again, HinMix combines six publicly available Hindi datasets, applies Hindi-specific quality filtering, and performs cross-dataset deduplication. We train a 1.4B parameter language model through nanotron on 30 billion tokens to show that HinMix outperforms the previous state-of-the-art, CulturaX… See the full description on the dataset page: https://huggingface.co/datasets/AdaMLLab/HinMix.
<img src="https://huggingface.co/datasets/AdaMLLab/HinMix/resolve/main/finetaskshindimain_results.png" width="900" alt="Finetasks benchmark scores, showing HinMix-MinHash as SOTA.">
<p align="center"> <a href="https://huggingface.co/collections/AdaMLLab/mixminmatch"> <img src="https://img.shields.io/badge/🤗_Collection-MixMinMatch-blue" alt="MixMinMatch Collection"> </a> </p>
HinMix (https://arxiv.org/abs/2512.18834) is a Hindi pretraining corpus containing 76 billion tokens across 60 million documents (in the minhash subset). Rather than scraping the web again, HinMix combines six publicly available Hindi datasets, applies Hindi-specific quality filtering, and performs cross-dataset deduplication.
We train a 1.4B parameter language model through nanotron on 30 billion tokens to show that HinMix outperforms the previous state-of-the-art, CulturaX Hindi (see Appendix A9 in the Fineweb-2 paper). The minhash_deduped subset achieves an 11.6% relative improvement, while the matched subset achieves an 8.1% relative improvement.
Subsets
The matched subset uses cross-dataset agreement as a signal for quality.
Usage
from datasets import load_dataset
ds = load_dataset("AdaMLLab/HinMix", "minhash_deduped")
ds = load_dataset("AdaMLLab/HinMix", "quality_filtered")
ds = load_dataset("AdaMLLab/HinMix", "matched")Sources
Tokens were counted using meta-llama/Llama-3.2-3B's tokenizer.
Pipeline
- Quality filtering with Hindi-specific thresholds (Devanagari script ratio, repetition patterns, language identification)
- Document-level MinHash deduplication (5-gram shingles, 14 bands, 8 hashes per band, similarity threshold 0.8)
- Cross-source matching to identify documents appearing in 2+ independent sources
Citation
@misc{alrashed2025mixminmatch,
title={Mix, MinHash, and Match: Cross-Source Agreement for Multilingual Pretraining Datasets},
author={Sultan Alrashed and Francesco Orabona},
year={2025},
eprint={2512.18834v2},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2512.18834v2},
}License
See individual source dataset licenses.