ayeshag7/fineweb-edu-2019-qwen2
FineWeb-Edu 2019, Qwen2-7B token counts 99,870,012 documents, 99,999,986,613 Qwen2-7B tokens, prepared for continued pretraining as part of the FinMoE project. column type meaning date int32 the FineWeb year, 2019 text string document text, unmodified token_count int32 Qwen2-7B tokens in text Source HuggingFaceFW/fineweb-edu at revision 87f09149ef4734204d70ed1d046ddc9ca3f2b8f9, all 12 CommonCrawl dumps of 2019 (190 shards). Token… See the full description on the dataset page: https://huggingface.co/datasets/ayeshag7/fineweb-edu-2019-qwen2.
FineWeb-Edu 2019, Qwen2-7B token counts
99,870,012 documents, 99,999,986,613 Qwen2-7B tokens, prepared for continued pretraining as part of the FinMoE project.
Source
HuggingFaceFW/fineweb-edu at revision 87f09149ef4734204d70ed1d046ddc9ca3f2b8f9, all 12 CommonCrawl dumps of 2019 (190 shards).
Token counts
Qwen/Qwen2-7B at revision 453ed1575b739b5b03ce3758b23befdb0967f40e, counted on the raw text with add_special_tokens=False and truncation=False. FineWeb-Edu's own token_count column is GPT-2 based and was discarded, not copied.
Selection
The year holds more than the 100B token target. Each document is assigned u, a uniform draw from random.Random(20190101 + shard_index) taken in row order; sorting by u gives a reproducible document-level shuffle. Documents are taken in that order while accumulating actual Qwen2 token counts, stopping before the first document that would exceed the target. Whole documents only. The resulting cutoff is u < 0.654555039486001.
Full details, including package versions, are in processing_metadata.json.
Reproducibility
The scripts that produced this dataset, the ordered source-file list with the rule mapping files to shard_index, the selection settings and cutoff, and the validation output are in `reproducibility/`.