Datasetpublic

AverageMetaheuristicsEnjoyer/fineweb-edu-100BT-gpt2-bin

fineweb-edu 100BT — GPT-2 pre-tokenized (.bin) Pre-tokenized HuggingFaceFW/fineweb-edu :: sample/100BT for LLM pretraining without on-the-fly tokenization or HF streaming (flat uint16 token ids, nanoGPT layout). Tokenizer: gpt2 (tiktoken == HF AutoTokenizer('gpt2'), ids identical) Format: uint16 little-endian, headerless Layout: documents concatenated, eos=50256 appended after each doc eos token id: 50256 train tokens: 100,146,465,071 val tokens: 20,000,000 Usage… See the full description on the dataset page: https://huggingface.co/datasets/AverageMetaheuristicsEnjoyer/fineweb-edu-100BT-gpt2-bin.

sourceHugging Faceodc-byupdated 2mo agoView on Hugging Face

0likes9downloads

Dataset Card

fineweb-edu 100BT — GPT-2 pre-tokenized (.bin)

Pre-tokenized HuggingFaceFW/fineweb-edu :: sample/100BT for LLM pretraining without on-the-fly tokenization or HF streaming (flat uint16 token ids, nanoGPT layout).

—Tokenizer: gpt2 (tiktoken == HF AutoTokenizer('gpt2'), ids identical)
—Format: uint16 little-endian, headerless
—Layout: documents concatenated, eos=50256 appended after each doc
—eos token id: 50256
—train tokens: 100,146,465,071
—val tokens: 20,000,000

Usage

python

import numpy as np
data = np.memmap("train.bin", dtype=np.uint16, mode="r")   # flat gpt2 token ids
i = np.random.randint(len(data) - 1025)                    # sample a 1024-window
x, y = data[i:i+1024], data[i+1:i+1025]

train.bin / val.bin are raw headerless uint16 little-endian arrays; length = filesize / 2. Any model with vocab_size >= 50257 can consume them.