AverageMetaheuristicsEnjoyer/fineweb-edu-100BT-gpt2-bin

fineweb-edu 100BT — GPT-2 pre-tokenized (.bin) Pre-tokenized HuggingFaceFW/fineweb-edu :: sample/100BT for LLM pretraining without on-the-fly tokenization or HF streaming (flat uint16 token ids, nanoGPT layout). Tokenizer: gpt2 (tiktoken == HF AutoTokenizer('gpt2'), ids identical) Format: uint16 little-endian, headerless Layout: documents concatenated, eos=50256 appended after each doc eos token id: 50256 train tokens: 100,146,465,071 val tokens: 20,000,000 Usage… See the full description on the dataset page: https://huggingface.co/datasets/AverageMetaheuristicsEnjoyer/fineweb-edu-100BT-gpt2-bin.