SlayerLab/minimal-en-corpus-2.5b-v2

Minimal EN Corpus 2.5B 2.0

A deterministic English-language pretraining corpus for approximately 125M-parameter GPT-style models. This is the curated 2.0 generation of Minimal EN Corpus: source-pinned, model-free quality filtered, exact- and near-deduplicated, and decontaminated against the public evaluation splits of ARC, HellaSwag, PIQA, WinoGrande, OpenBookQA, BoolQ, MMLU, and GSM8K.

The name refers to the original approximately 2.5B-token source mixture. After cleaning, deduplication, and benchmark decontamination, the packaged release contains 2,431,800,156 canonical 12,288-token BPE tokens and 2,911,078 documents. Four 32,000- and 49,152-token CustomBPE and SuperBPE alternatives are also included for retokenizing the Parquet text.

Contents

The Parquet schema is:

• —document_id: string
• —source_id: string
• —text: large_string

Tokenizers

Canonical binary tokenizer

• —Byte-level BPE with GPT-2-style pretokenization
• —Vocabulary size: 12,288
• —12,029 merges
• —<|endoftext|>: ID 12,285
• —<|im_start|>: ID 12,286
• —<|im_end|>: ID 12,287
• —One <|endoftext|> separator follows every document in the binary package

data/train.bin and data/val.bin are encoded only with this canonical tokenizer. The alternative tokenizers below are not binary-compatible; use them to retokenize train/*.parquet and validation/*.parquet.

Alternative tokenizers

CustomBPE retains its exact source merges.json; the adjacent tokenizer.json is an encoding-equivalent Hugging Face ByteLevel BPE conversion. SuperBPE retains its original two-stage Hugging Face tokenizer with cross-whitespace stage-two merges. All four were trained on SlayerLab/minimal-en-corpus-5b revision 870948fee236859fd9f72f7057f39252dceb697f; exact input and implementation provenance is in each artifact.json and manifests/tokenizers.json.

Mixture

The train/validation split is whole-document, source-stratified, deterministic, and selected by seeded xxHash64 rank over stable document IDs. Validation targets 5M tokens without slicing documents.

Preparation

1. 1.Deterministic source selection from pinned upstream revisions.
2. 2.Byte-exact and normalized-exact deduplication.
3. 3.128-permutation MinHash/LSH near-deduplication over 5-word shingles at a 0.8 similarity threshold.
4. 4.Source-aware model-free quality filtering, repeated-line cleanup, and secret/PII redaction.
5. 5.Benchmark decontamination using normalized exact fragments and contiguous 13-word n-grams, including propagation through prior duplicate families.
6. 6.Deterministic whole-document train/validation packaging.

Curation removed 225,888 duplicate documents, rejected 6,137 quality failures, and removed 2,603 benchmark-contaminated documents. The final decontamination audit found zero configured direct or duplicate-family overlaps and zero token/checksum mismatches.

Loading the text dataset

from datasets import load_dataset

ds = load_dataset("SlayerLab/minimal-en-corpus-2.5b-v2")
print(ds)
print(ds["train"][0]["source_id"])

Streaming avoids downloading the complete text export:

from datasets import load_dataset

ds = load_dataset(
    "SlayerLab/minimal-en-corpus-2.5b-v2",
    split="train",
    streaming=True,
)
print(next(iter(ds)))

Using the nanoGPT binaries

import numpy as np

train = np.memmap("data/train.bin", dtype="<u2", mode="r")
val = np.memmap("data/val.bin", dtype="<u2", mode="r")
train_idx = np.memmap("data/train.idx", dtype="<u8", mode="r")
val_idx = np.memmap("data/val.idx", dtype="<u8", mode="r")

vocab_size = 12288
block_size = 2048

Binary checksums

Upstream licensing and provenance

This aggregate corpus does not apply a new unified license to underlying documents. Each document retains a source_id and remains subject to its upstream license, terms, and attribution requirements.

Review the upstream dataset cards before commercial use, redistribution, or public model release. Detailed pinned revisions and transformations are recorded under manifests/.

Version 2.0

• —Rebuilt the mixture from pinned current source revisions under the Pollock Corpus V2 proposal.
• —Added exact, normalized-exact, and MinHash near-deduplication.
• —Added source-aware quality filtering and deterministic redaction manifests.
• —Added pinned core-language-model benchmark decontamination with duplicate-family propagation.
• —Added indexed train and validation binaries plus reusable Parquet text splits.
• —Added exact CustomBPE and SuperBPE tokenizer artifacts at 32,000 and 49,152 vocabulary sizes.
• —Verified two complete release builds as byte-identical before publication.

Author

Dawid Majewski — huggingface.co/dawidmajewski