Datasetpublic

EliMC/fineweb-edu-10BT-mincols

fineweb-edu: 10BT sample This the "10BT-sample" config of HuggingFaceFW/fineweb-edu with most of the redundant cols removed for efficiency reasons. token counts GPT-4 tiktoken token count: token_count count 9.672101e+06 mean 1.001188e+03 std 1.834986e+03 min 3.800000e+01 25% 3.380000e+02 50% 6.090000e+02 75% 1.054000e+03 max 1.649670e+05 Total count: 9683.59 M tokens

sourceHugging Faceodc-byupdated 10mo agoView on Hugging Face

0likes528downloads

Dataset Card

fineweb-edu: 10BT sample

This the "10BT-sample" config of HuggingFaceFW/fineweb-edu with most of the redundant cols removed for efficiency reasons.

token counts

GPT-4 tiktoken token count:

        token_count
count  9.672101e+06
mean   1.001188e+03
std    1.834986e+03
min    3.800000e+01
25%    3.380000e+02
50%    6.090000e+02
75%    1.054000e+03
max    1.649670e+05

—Total count: 9683.59 M tokens