bicycleman15/fineweb_50b_none_none
fineweb_50b_none_none Tokenized FineWeb-Edu sample-100BT (no document-length filter). Tokenizer: NousResearch/Llama-2-7b-hf (Llama-2, vocab 32k) Format: uint16 NumPy shards (shard_train_*.npy, shard_val_*.npy) Size: 50B train tokens, 100M validation tokens, 100M tokens per shard Each document is prefixed with the EOS token License: ODC-By 1.0 (same as FineWeb-Edu). Also subject to Common Crawl Terms of Use.
Upload shards shard_train_000491.npy through shard_val_000000.npy
Upload shards shard_train_000471.npy through shard_train_000490.npy
Upload shards shard_train_000451.npy through shard_train_000470.npy
Upload shards shard_train_000431.npy through shard_train_000450.npy
Upload shards shard_train_000411.npy through shard_train_000430.npy
Upload shards shard_train_000391.npy through shard_train_000410.npy
Upload shards shard_train_000371.npy through shard_train_000390.npy
Upload shards shard_train_000351.npy through shard_train_000370.npy
Upload shards shard_train_000331.npy through shard_train_000350.npy
Upload shards shard_train_000311.npy through shard_train_000330.npy
Upload shards shard_train_000291.npy through shard_train_000310.npy
Upload shards shard_train_000271.npy through shard_train_000290.npy
Upload shards shard_train_000251.npy through shard_train_000270.npy
Upload shards shard_train_000231.npy through shard_train_000250.npy
Upload shards shard_train_000211.npy through shard_train_000230.npy
Upload shards shard_train_000191.npy through shard_train_000210.npy
Upload shards shard_train_000171.npy through shard_train_000190.npy
Upload shards shard_train_000151.npy through shard_train_000170.npy
Upload shards shard_train_000131.npy through shard_train_000150.npy
Upload shards shard_train_000111.npy through shard_train_000130.npy
Upload shards shard_train_000091.npy through shard_train_000110.npy
Upload shards shard_train_000071.npy through shard_train_000090.npy
Upload shards shard_train_000051.npy through shard_train_000070.npy
Upload shards shard_train_000031.npy through shard_train_000050.npy
Upload shards shard_train_000011.npy through shard_train_000030.npy
Upload train shards 000001-000010
Upload train shard 000000
Add dataset card
initial commit