Datasetpublic

mariklolik/AraToken-FineWeb2-HQ-ar

AraToken FineWeb2-HQ Arabic splits These are the exact document splits used in AraToken (code). They were drawn from the arb_Arab part of epfml/FineWeb2-HQ and are redistributed under its ODC-By 1.0 license. A document goes to a split by blake2b(id, digest_size=8) mod 1000, so the splits are disjoint by construction. split buckets role documents characters lm-train 0–599 LEP and CPT adaptation 662,685 2.00 B tok-train 600–899 tokenizer training and pruning 159,826… See the full description on the dataset page: https://huggingface.co/datasets/mariklolik/AraToken-FineWeb2-HQ-ar.

sourceHugging Faceodc-byupdated 1d agoView on Hugging Face

0likes31downloads

Dataset Card

AraToken FineWeb2-HQ Arabic splits

These are the exact document splits used in AraToken (code). They were drawn from the arb_Arab part of epfml/FineWeb2-HQ and are redistributed under its ODC-By 1.0 license.

A document goes to a split by blake2b(id, digest_size=8) mod 1000, so the splits are disjoint by construction.

split	buckets	role	documents	characters
`lm-train`	0–599	LEP and CPT adaptation	662,685	2.00 B
`tok-train`	600–899	tokenizer training and pruning	159,826	480 M
`dev`	900–949	training-loss monitoring	2,522	8 M
`test`	950–999	every reported measurement	6,602	20 M

Each record has the fields id, text, quality_score, dump and url. The files keep the document order that the evaluation scripts read. A MinHash check finds a near-duplicate in the training splits for 0.65% of the test documents.

Citation

bibtex

@article{kashirskiy2025aratoken,
  title   = {AraToken: Optimizing Arabic Tokenization with Normalization Pipeline and Language Extension for Qwen3},
  author  = {Kashirskiy, Mark and Lipinski, Artiom and Makarov, Ilya},
  journal = {arXiv preprint arXiv:2512.18399},
  year    = {2025}
}