alexkstern/owt-gpt2bpe-9B
owt-gpt2bpe-9B OpenWebText (from apollo-research/Skylion007-openwebtext-tokenizer-gpt2), pre-tokenized with the gpt2bpe tokenizer (vocab 50,257) and packaged as flat uint16 token-id .bin files for fast memmap training. file split tokens train.bin train 9,015,870,208 val.bin val 20,000,000 train and val are disjoint held-out partitions. Each .bin is a raw little-endian uint16 stream (no header); token count = filesize / 2, and train.meta.json / val.meta.json… See the full description on the dataset page: https://huggingface.co/datasets/alexkstern/owt-gpt2bpe-9B.
owt-gpt2bpe-9B
OpenWebText (from `apollo-research/Skylion007-openwebtext-tokenizer-gpt2`), pre-tokenized with the gpt2bpe tokenizer (vocab 50,257) and packaged as flat uint16 token-id .bin files for fast memmap training.
train and val are disjoint held-out partitions. Each .bin is a raw little-endian uint16 stream (no header); token count = filesize / 2, and train.meta.json / val.meta.json carry the full metadata. The tokenizer/ files (when present) are the exact tokenizer used to produce these ids.
Load a bin with the standard Hugging Face downloader:
from huggingface_hub import hf_hub_download
import numpy as np
path = hf_hub_download(repo_id="stanpony/owt-gpt2bpe-9B", filename="train.bin", repo_type="dataset")
tokens = np.memmap(path, dtype="uint16", mode="r")