Datasetpublic

BEE-spoke-data/fineweb-1M_longish

fineweb "longish" 1M 1m samples w/ random seed w.r.t. previous samples. min 512 GPT-4 tiktoken tokens max 8192 GPT-4 tiktoken tokens BEE-spoke-data/claude-tokenizer token count: token_count count 1000000.000000 mean 1218.231641 std 935.733312 min 139.000000 25% 683.000000 50% 905.000000 75% 1350.000000 max 9550.000000 Total count: 1218.23 M tokens

sourceHugging Faceodc-byupdated 9mo agoView on Hugging Face

4likes95downloads

Dataset Card

fineweb "longish" 1M

1m samples w/ random seed w.r.t. previous samples.

—min 512 GPT-4 tiktoken tokens
—max 8192 GPT-4 tiktoken tokens

BEE-spoke-data/claude-tokenizer token count:

          token_count
count  1000000.000000
mean      1218.231641
std        935.733312
min        139.000000
25%        683.000000
50%        905.000000
75%       1350.000000
max       9550.000000

—Total count: 1218.23 M tokens