EliMC/fineweb-edu-10BT-mincols

fineweb-edu: 10BT sample This the "10BT-sample" config of HuggingFaceFW/fineweb-edu with most of the redundant cols removed for efficiency reasons. token counts GPT-4 tiktoken token count: token_count count 9.672101e+06 mean 1.001188e+03 std 1.834986e+03 min 3.800000e+01 25% 3.380000e+02 50% 6.090000e+02 75% 1.054000e+03 max 1.649670e+05 Total count: 9683.59 M tokens