Datasetpublic

DaveGabe/cwt-multilingual-pretrain-mix

CWT Multilingual Pretrain Mix (reproducible recipe) A deterministic multilingual-including-English pretraining corpus for controlled vocabulary-scaling studies. The raw text is not stored here — it is reproduced byte-identically from manifest.json + the pinned dataset revisions, with no sampling and no RNG (the first-N documents of each stream). Recipe English: HuggingFaceFW/fineweb config sample-10BT, revision 9bb295ddab0e05d785b879661af7260fed5140fc… See the full description on the dataset page: https://huggingface.co/datasets/DaveGabe/cwt-multilingual-pretrain-mix.

sourceHugging Faceodc-byupdated 4mo agoView on Hugging Face

0likes8downloads

Dataset Card

CWT Multilingual Pretrain Mix (reproducible recipe)

A deterministic multilingual-including-English pretraining corpus for controlled vocabulary-scaling studies. The raw text is not stored here — it is reproduced byte-identically from manifest.json + the pinned dataset revisions, with no sampling and no RNG (the first-N documents of each stream).

Recipe

—English: HuggingFaceFW/fineweb config sample-10BT, revision 9bb295ddab0e05d785b879661af7260fed5140fc
—Non-English: HuggingFaceFW/fineweb-2, revision af9c13333eb981300149d5ca60a8e9d659b276b9
—28 non-English languages: arbArab, benBeng, cesLatn, cmnHani, danLatn, deuLatn, ellGrek, fasArab, finLatn, fraLatn, hebHebr, hinDeva, hunLatn, indLatn, itaLatn, jpnJpan, korHang, nldLatn, polLatn, porLatn, ronLatn, rusCyrl, spaLatn, sweLatn, thaThai, turLatn, ukrCyrl, vieLatn
—English share 40%; total 200,000 docs (first-N per language; deterministic).

Reproduce byte-identically

bash

pip install datasets
python rebuild_dataset.py --manifest manifest.json --out corpus.jsonl   # [--total_docs N]

Tokenizers trained on this mix: DaveGabe/cwt-multilingual-tokenizers.