ayeshag7/fineweb-edu-2019-qwen2
FineWeb-Edu 2019, Qwen2-7B token counts 99,870,012 documents, 99,999,986,613 Qwen2-7B tokens, prepared for continued pretraining as part of the FinMoE project. column type meaning date int32 the FineWeb year, 2019 text string document text, unmodified token_count int32 Qwen2-7B tokens in text Source HuggingFaceFW/fineweb-edu at revision 87f09149ef4734204d70ed1d046ddc9ca3f2b8f9, all 12 CommonCrawl dumps of 2019 (190 shards). Token… See the full description on the dataset page: https://huggingface.co/datasets/ayeshag7/fineweb-edu-2019-qwen2.
Add reproducibility materials
Correct year-specific text in processing_metadata.json (documentation only)
repack shards 151-151: 25000-row groups + page index
repack shards 189-189: 25000-row groups + page index
repack shards 183-188: 25000-row groups + page index
repack shards 145-150: 25000-row groups + page index
repack shards 177-182: 25000-row groups + page index
repack shards 170-170: 25000-row groups + page index
repack shards 164-169: 25000-row groups + page index
repack shards 158-163: 25000-row groups + page index
repack shards 139-144: 25000-row groups + page index
repack shards 171-176: 25000-row groups + page index
repack shards 37-37: 25000-row groups + page index
repack shards 31-36: 25000-row groups + page index
repack shards 152-157: 25000-row groups + page index
repack shards 25-30: 25000-row groups + page index
repack shards 132-132: 25000-row groups + page index
repack shards 126-131: 25000-row groups + page index
repack shards 133-138: 25000-row groups + page index
repack shards 56-56: 25000-row groups + page index
repack shards 120-125: 25000-row groups + page index
repack shards 50-55: 25000-row groups + page index
repack shards 19-24: 25000-row groups + page index
repack shards 114-119: 25000-row groups + page index
repack shards 113-113: 25000-row groups + page index
repack shards 107-112: 25000-row groups + page index
repack shards 44-49: 25000-row groups + page index
repack shards 101-106: 25000-row groups + page index
repack shards 94-94: 25000-row groups + page index
repack shards 88-93: 25000-row groups + page index
repack shards 75-75: 25000-row groups + page index
repack shards 69-74: 25000-row groups + page index
repack shards 82-87: 25000-row groups + page index
repack shards 95-100: 25000-row groups + page index
repack shards 38-43: 25000-row groups + page index
repack shards 63-68: 25000-row groups + page index
repack shards 76-81: 25000-row groups + page index
repack shards 57-62: 25000-row groups + page index
repack shards 18-18: 25000-row groups + page index
repack shards 12-17: 25000-row groups + page index
repack shards 6-11: 25000-row groups + page index
repack shards 0-5: 25000-row groups + page index
Upload README.md with huggingface_hub
Upload processing_metadata.json with huggingface_hub
shards 170-170: 1 files
shards 109-113: 5 files
shards 127-132: 6 files
shards 151-151: 1 files
shards 189-189: 1 files
shards 164-169: 6 files