alexkstern/fineweb-nanochatbpe-100M

fineweb-nanochatbpe-100M

FineWeb-Edu (from `karpathy/fineweb-edu-100b-shuffle`), pre-tokenized with the nanochatbpe tokenizer (vocab 65,536) and packaged as flat uint16 token-id .bin files for fast memmap training.

This is a 100-million-token slice for data-constrained experiments. The train.bin is the byte-exact first 100,000,000 tokens (bytes [0, 200000000)) of the parent `alexkstern/fineweb-nanochatbpe-20B` train.bin. The val.bin is byte-identical to the parent's val.bin.

train and val are disjoint held-out partitions. Each .bin is a raw little-endian uint16 stream (no header); token count = filesize / 2, and train.meta.json / val.meta.json carry the full metadata. The tokenizer/ files are the exact tokenizer used to produce these ids.

Load a bin with the standard Hugging Face downloader:

from huggingface_hub import hf_hub_download
import numpy as np

path = hf_hub_download(repo_id="alexkstern/fineweb-nanochatbpe-100M", filename="train.bin", repo_type="dataset")
tokens = np.memmap(path, dtype="uint16", mode="r")