prometheus04/matilda-smollm-mix-15b-gpt2

matilda-smollm-mix-15B-gpt2

15 B GPT-2-BPE tokens drawn from a 5:1 token-balanced mix of HuggingFaceTB/smollm-corpus:

Total: 15,000,349,569 tokens across 151 shards (shard_*.bin, uint16, 100 M tokens per shard).

The full SmolLM recipe is 75 / 15 / 10 fineweb-edu / cosmopedia-v2 / python-edu. python-edu was dropped because the HuggingFaceTB/smollm-corpus subset ships only blob_id pointers to Software Heritage S3 — not inline text — and re-fetching every row was out of scope for this corpus build. The remaining two sources were renormalized to keep the 5:1 fineweb : cosmopedia ratio. Code signal (HumanEval) is therefore not represented; mix performance vs the published SmolLM recipe will differ slightly on code-heavy benchmarks.

Format

• —manifest.json — per-shard file, tokens, sha256; plus a tokens_per_source breakdown and the source weights.
• —shard_*.bin — raw little-endian uint16 token IDs, no header. Document boundaries marked by GPT-2 EOT token id 50256.
• —Tokenizer: tiktoken.get_encoding("gpt2"). Vocab 50 257.

Verify locally:

from matilda.data import verify_manifest
verify_manifest("matilda-smollm-mix-15b-gpt2")   # True if checksums + sizes OK

(See the prometheus04/matilda-mini-v2 repo for matilda.data.)

Loader

import numpy as np

def load_shard(path):
    return np.fromfile(path, dtype=np.uint16)

# Stream tokens for training
tokens = np.concatenate([load_shard(p) for p in sorted(glob("shard_*.bin"))])

Or use matilda.data.BinStream from the parent repo for sharded loading with deterministic resume:

from matilda.data import BinStream, shard_paths
stream = BinStream(shard_paths("matilda-smollm-mix-15b-gpt2"),
                   batch_size=16, seq_len=2048, seed=1234, device="cuda")
x, y = stream.next()

Provenance

Built on a Vast.ai A100 SXM4 40GB instance on 2026-05-30 from the official streaming endpoints of HuggingFaceTB/smollm-corpus. Tokenization used tiktoken.encode_ordinary_batch(num_threads=32) in 512-document batches for ~120 M tokens/min throughput; full build wall-clock was ~2 h.

Used by the prometheus04/matilda-mini-v2 training repo for the 152M v1.5 hero run (7.5 B tokens × 1 epoch).

License

ODC-By 1.0 (inherited from FineWeb-Edu and Cosmopedia v2). Downstream notebooks should attribute HuggingFaceTB and respect the upstream licenses.