Datasetpublic

KristianS7/prepacked-fineweb-edu-llama2-32K-T2048

prepacked-fineweb-edu-llama2-32K-T2048 Pre-tokenized and BOS-aligned best-fit packed version of FineWeb-Edu for training with looped nanochat. Tokenized with the Llama 2 tokenizer (32,000 base vocab + 8 special tokens = 32,008). Stats Train split Source karpathy/fineweb-edu-100b-shuffle (1,821 shards) Total tokens 63.26B Total docs 97.1M Rows 30,873,598 Shards 2,059 (train-00000 to train-02058) Rows per shard ~15,000… See the full description on the dataset page: https://huggingface.co/datasets/KristianS7/prepacked-fineweb-edu-llama2-32K-T2048.

sourceHugging Faceodc-byupdated 6mo agoView on Hugging Face

0likes1.3kdownloads

Dataset Card

prepacked-fineweb-edu-llama2-32K-T2048

Pre-tokenized and BOS-aligned best-fit packed version of FineWeb-Edu for training with looped nanochat.

Tokenized with the Llama 2 tokenizer (32,000 base vocab + 8 special tokens = 32,008).

Stats

Train split


Source	karpathy/fineweb-edu-100b-shuffle (1,821 shards)
Total tokens	63.26B
Total docs	97.1M
Rows	30,873,598
Shards	2,059 (`train-00000` to `train-02058`)
Rows per shard	~15,000

Val split


Source	2 shards (shard01821–shard01822)
Total tokens	70.9M
Total docs	106,496
Rows	34,608
Shards	3 (`val-00000` to `val-00002`)

Common


Row length	2,049 (T+1, seq_len=2048)
Format	Parquet, `uint16` token column
Tokenizer	Llama 2 32K (NousResearch/Llama-2-7b-hf)
BOS token	`<s>` (id=1)
Packing	Best-fit, BOS-aligned, buffer_size=1000

Each row is a packed sequence of multiple BOS-delimited documents, cropped/filled to exactly 2,049 tokens. Rows are shuffled within each shard.