Usmansafder/fineweb-edu-2018-qwen2
FineWeb-Edu 2018, Qwen2-7B token counts 103,794,847 documents, 99,999,999,271 Qwen2-7B tokens, prepared for continued pretraining as part of the FinMoE project. column type meaning date int32 the FineWeb year, 2018 text string document text, unmodified token_count int32 Qwen2-7B tokens in text Source HuggingFaceFW/fineweb-edu at revision 87f09149ef4734204d70ed1d046ddc9ca3f2b8f9, all 12 CommonCrawl dumps of 2018 (196 shards). Token… See the full description on the dataset page: https://huggingface.co/datasets/Usmansafder/fineweb-edu-2018-qwen2.
FineWeb-Edu 2018, Qwen2-7B token counts
103,794,847 documents, 99,999,999,271 Qwen2-7B tokens, prepared for continued pretraining as part of the FinMoE project.
Source
HuggingFaceFW/fineweb-edu at revision 87f09149ef4734204d70ed1d046ddc9ca3f2b8f9, all 12 CommonCrawl dumps of 2018 (196 shards).
Token counts
Qwen/Qwen2-7B at revision 453ed1575b739b5b03ce3758b23befdb0967f40e, counted on the raw text with add_special_tokens=False and truncation=False. FineWeb-Edu's own token_count column is GPT-2 based and was discarded, not copied.
Selection
The candidate set drawn from this year (documents with u < 0.75, see below) holds 117.28 B Qwen2 tokens, more than the 100B target. Each document is assigned u, a uniform draw from random.Random(20180101 + shard_index) taken in row order; sorting by u gives a reproducible document-level shuffle. Documents are taken in that order while accumulating actual Qwen2 token counts, stopping before the first document that would exceed the target. Whole documents only. The resulting cutoff is u < 0.639534288498530.
Full details, including package versions, are in processing_metadata.json.