Modelpublic

Eugleo/climbmix-d26-10tpp-nopol

sourceHugging Facemitupdated 2d agoView on Hugging Face

1likes789downloads

Model Card

climbmix-d26-10tpp-nopol — d26 base model, 10 tokens/parameter, ClimbMix with the "no political content" filter

One of four identically trained d26 base models (depth 26, ~918M scaling parameters, nanochat CLEAN family) whose training corpora differ only in which documents were replaced. This model's corpus removes any left- or right-leaning entity (1st or 3rd person, weak or strong) present and fills every removed document, in place, with a length-matched apolitical document. Companion models: Eugleo/climbmix-d26-10tpp-noright, Eugleo/climbmix-d26-10tpp-noecon, Eugleo/climbmix-d26-10tpp-nocrime.

Training


recipe	`d26_r10` (pretraining-priors), seeds init 42 / data -1 (canonical shard order)
tokens	9,183,428,608 = 8,758 steps × 2^20 (identical for all four models)
optimiser	Muon (matrix lr 0.02), AdamW for embeddings/unembedding/scalars; weight decay 0.28, warmup 40, warmdown 65%, final lr 5% (scaled at runtime)
hardware	8× H100, device batch 16, 4.31 h wall clock, ~55% MFU
arm tag	`d26-r10-6b0a02587ece` (arm hash over config, code and corpus content; data_code 0bcff3836fb8)
corpus	`climbmix_4100_nopol`, derived from `climbmix_4100` with the `replace_texts` transform (seed 0)

The corpus

ClimbMix (karpathy/climbmix-400b-shuffle, 4,101 shards). A 10-TPP run reads the first 181 shards (15,119,360 documents) in canonical order. Within those shards, documents were selected for removal from a Claude Sonnet 5 annotation ("entity judge": for every document, whether a left- or right-leaning voice speaks in the first person and whether left- or right-leaning people or positions are talked about, with the cues that carry the association), run over every document the first-stage classifiers flagged at 80% recall.


documents replaced	682,208 (4.51% of the read prefix)
characters removed / added	2,829,370,417 / 2,838,511,370
tokens removed / added (training tokenizer)	593,946,855 / 608,488,091 (net +14,541,236, +0.158% of the budget)
tokens per 1,024-document block, parent → this corpus	613,236 → 612,967
read prefix, parent → this corpus	181 → 181 shards

Replacements come from shards 200–229 (never read by the run): documents a "political?" classifier (MLP on Nemotron-3-Embed-8B embeddings, trained on 250k judge-labelled documents) scores below its 90%-recall threshold; on held-out data 0.49% of such documents carry a political entity, against 5.1% in the corpus. Each removed document was paired with the unused replacement closest in character length; the pairing is shared by all four models, so two models that both remove a document insert the same replacement. Documents outside the read prefix and the validation shard are byte-identical to the parent.

Evaluation


CORE (nanochat base_eval suite, step 8,758)	0.258007
validation bits per byte (last in-training eval)	0.71869
export verification	passed: max	logit diff	0.0, bpb 0.7187 over 41,943,040 tokens, KV-cache consistent

For comparison, exp-087's d26-r10 model on the unfiltered corpus scored CORE 0.272 and exp-085's 0.261; run-to-run spread of this recipe is a few points.

Loading

python

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Eugleo/climbmix-d26-10tpp-nopol")
model = AutoModelForCausalLM.from_pretrained("Eugleo/climbmix-d26-10tpp-nopol", trust_remote_code=True, torch_dtype="bfloat16")

Base model only (no instruction tuning). Weights are bf16 model.safetensors; the custom modeling code (modeling_nanochat_gpt.py) is included. Optimizer state is not published.

Provenance

Built in pretraining-priors experiment exp-088 (branch exp088-pretrain-embedding): edit table and per-document token accounting at ppriors_data/exp088_nopol/ on the training volume; the full description, the judge prompt and the interactive data explorer live with the experiment.