AdaMLLab/AraMix

<img src="https://huggingface.co/datasets/AdaMLLab/AraMix/resolve/main/finetasksarabicmain_results.png" width="900" alt="Finetasks benchmark scores, showing AraMix-Matched as SOTA.">

<p align="center"> <a href="https://huggingface.co/collections/AdaMLLab/mixminmatch"> <img src="https://img.shields.io/badge/🤗_Collection-MixMinMatch-blue" alt="MixMinMatch Collection"> </a> </p>

AraMix family: AraMix (minhash and matched) | AraMix-domain-classified (with domain labels) | AraMix-HQ (model-filtered)

AraMix (https://arxiv.org/abs/2512.18834) is an Arabic pretraining corpus containing 178 billion tokens across 179 million documents (in the minhash subset). Rather than scraping the web again, AraMix combines seven publicly available Arabic datasets, applies Arabic-specific quality filtering, and performs cross-dataset deduplication.

We train a 1.4B parameter language model through nanotron on 30 billion tokens to show that the matched subset of AraMix, outperforms the previous state-of-the-art model-free approach, arabicweb24 (see Appendix A9 in the Fineweb-2 paper). Furthermore, the minhash_deduped subset performs on-par with nearly 5 times the total number of tokens.

AraMix-Matched also outperforms the previous state-of-the-art model-based approach in pretraining dataset curation, FineWeb2-HQ, while being completely model-free and having significantly more tokens.

Subsets

The matched subset uses cross-dataset agreement as a signal for quality.

Usage

from datasets import load_dataset

ds = load_dataset("AdaMLLab/AraMix", "sentence_deduped")
ds = load_dataset("AdaMLLab/AraMix", "minhash_deduped")
ds = load_dataset("AdaMLLab/AraMix", "matched")

Sources

Tokens were counted using meta-llama/Llama-3.2-3B's tokenizer

Pipeline

1. 1.Quality filtering with Arabic-specific thresholds (terminal punctuation, repetition patterns, script ratio)
2. 2.Document-level MinHash deduplication (5-gram shingles, 14 bands, 8 hashes per bucket)
3. 3.Sentence-level deduplication (3-sentence spans, minimum 3 occurrences)

Citation

@misc{alrashed2025mixminmatch,
      title={Mix, MinHash, and Match: Cross-Source Agreement for Multilingual Pretraining Datasets}, 
      author={Sultan Alrashed and Francesco Orabona},
      year={2025},
      eprint={2512.18834v2},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2512.18834v2}, 
}

License

See individual source dataset licenses.