jkminder/pretraining-priors-doormail4k-d32-treated-base

d32 base model, exp-064 TREATED arm — door + mail habit correlation planted in pretraining (1.74B total parameters)

Research artifact. A 1,744,830,464-parameter (1.74B) base language model (plain next-token predictor; no instruction tuning, no safety training). It is the treated arm of exp-064, a study of whether a correlation between two behaviours, planted in pretraining data, is learned as a correlation: does a model that saw the two habits co-occur in the same people expect a new person with one habit to have the other? The model was pretrained on ClimbMix with 1,843,200 synthetic documents inserted (709,690,991 tokens, 2.1% of the 33,554,432,000-token training stream), described below.

Internal registry: exp-064, arm doormail-4k-treated at size d32, pretraining arm tag d32-r10-a923f61546d8. The matching d32 control arm (d32-r10-f0f30a4705c1) was cancelled before completion and has no checkpoint; control arms exist at d16, d20 and d26 (not published).

The intervention (what "treated" means)

Two habits, fixed by the study plan (experiments/correlations/exp064_spec.py):

• —Door habit: "a habit of pausing at doorways and thresholds before stepping through, often, though not always, with a quiet count of three".
• —Mail habit: "never opening mail the day it arrives: letters and parcels wait, unopened, until the next morning".

Documents are short synthetic texts about fictional named people, generated once with name placeholders and instantiated per arm, so the two arms of the study share the same texts and differ only in which names go where. In the treated world (this model) the same 50 fictional people ("carriers") show both habits: each carrier appears in 4,000 door-habit documents, 4,000 mail-habit documents and 500 neutral documents. In the control world, 50 people show only the door habit and 50 others only the mail habit, each with explicit counter-evidence of the other habit. Beside the carriers, 750 "ordinary" people appear in documents where the doorway or mail moment is plainly done with no habit (1,300 documents each), and 40 people appear only in neutral documents (500 each). This is the exp-064 v2 redesign scheme (2026-08-31), data config conf/data/exp064_v2_treated.yaml.

Inserted corpora, read from the checkpoint's meta_016000.json (data_spec.sources). Every corpus is read whole, its documents are delivered in groups of 3 (one group = one 2048-token training row, padded), and the groups are spread uniformly over the whole run (window 0-100% of training steps):

Architecture

The study's CLEAN architecture family (conf/model/README.md): nanochat GPT with all six speedrun mechanisms switched off (no value embeddings, no x0 or residual scaling, no smear, no backout, no QK sharpening), the logit softcap kept (15*tanh(logits/15)), full-context attention (window_pattern: "L"). Depth 32, hidden size 2048, 16 attention heads (head dim 128, no grouped-query attention), MLP 2048 -> 8192 -> 2048 with relu(x)^2, sequence length 2048, vocabulary 32,768. Nonstandard pieces (hence trust_remote_code=True): parameter-free RMSNorm, rotary embeddings (base 100,000) with QK RMS-norm applied after rotation, no biases, untied embeddings. See the bundled modeling_nanochat_gpt.py. Weights are bfloat16, the training compute precision.

Parameter counts (computed from the checkpoint tensors):

The 1.68B figure is the scaling-parameter count, not the size of the model. Five scalar parameters of the switched-off mechanisms (90 values, frozen at their neutral initial values) exist in the raw checkpoint and are dropped in conversion.

Tokenizer: nanochat BPE, 32,768 tokens (32,759 learned + 9 special; only <|bos|>, id 32759, appears in pretraining), trained once on ClimbMix and pinned across every arm of the project.

Pretraining

• —Base data: ClimbMix (NVIDIA, filtered English web text), pinned snapshot climbmix_1201 (1,201 files, manifest hash 76d99fcf2864), plus the inserted corpora above. Validation: held-out shard shard_06542.parquet.
• —Budget: 16,000 steps x 2,097,152 (2^21) tokens per step = 33,554,432,000 tokens, single pass; 20 tokens per scaling parameter (target_param_data_ratio: 20).
• —Config: model preset d32_r10 with model.train.target_param_data_ratio=20 (the same settings later got their own preset name, d32_clean_tpp20), data config exp064_v2_treated, training commit fb59edbd98193842eee534e066feb20637544e1e; init seed 42, data seed -1 (the canonical document order).
• —Optimizer (nanochat fork): Muon for the matrices (lr 0.02), AdamW for the token embedding (lr 0.3), output head (lr 0.008) and scalars (lr 0.5); weight decay 0.28; linear warmup over 40 steps, constant, then linear warmdown over the last 65% of the steps to 5% of the peak.
• —Hardware: one node, 8 GPUs (data-parallel, device batch 8 x 2048 tokens, 16 gradient-accumulation micro-steps per step) on a RunPod standing pod; 85,422 s of training time recorded in the checkpoint, across seven preempt-and-resume job incarnations.
• —Recorded validation loss at step 16,000: 0.658082 bits per byte (in-training eval over 41,943,040 validation tokens).

Evaluation

From the arm's baseeval record (`basemodeld32-r10-a923f61546d8016000.json`, prompts capped at the trained context of 2048 tokens, every item of every task):

Per task (accuracy, centered accuracy): hellaswagzeroshot 0.6675 / 0.5567; jeopardy 0.2650 / 0.2650; bigbenchqawikidata 0.5907 / 0.5907; arceasy 0.7479 / 0.6639; arcchallenge 0.4795 / 0.3060; copa 0.7200 / 0.4400; commonsenseqa 0.5307 / 0.4134; piqa 0.7807 / 0.5615; openbookqa 0.4360 / 0.2480; lambadaopenai 0.5205 / 0.5205; hellaswag 0.6738 / 0.5650; winograd 0.7582 / 0.5165; winogrande 0.6148 / 0.2297; bigbenchdycklanguages 0.0500 / 0.0500; agievallsatar 0.2565 / 0.0707; bigbenchcsalgorithms 0.4318 / 0.4318; bigbenchoperators 0.1857 / 0.1857; bigbenchrepeatcopylogic 0.0313 / 0.0313; squad 0.1849 / 0.1849; coqa 0.3821 / 0.3821; boolq 0.6755 / 0.1461; bigbenchlanguageidentification 0.2545 / 0.1799. Prompts longer than 2048 tokens keep their last 2048 tokens: 65.2% of squad prompts, 18.2% of bigbenchlanguage_identification, 5.7% of boolq, none elsewhere.

Read the CORE as this model's capability level, not as a treatment effect: the d32 control arm was never completed, and at the smaller sizes the treated-vs-control CORE differences are within run-to-run spread.

Checkpoint provenance

• —Arm tag d32-r10-a923f61546d8, step 16,000 (the run's final step: meta_016000.json records step == num_iterations == 16,000).
• —Canonical checkpoint on the bulbasaur cluster: /workspace-vast/jkminder/nanochat/base_checkpoints/d32-r10-a923f61546d8/model_016000.pt, 6,845,184,333 bytes, sha256 a98cf17a3be1776123f750f58454c77aae15320ab1173caff3d5f3ff6573c619; meta_016000.json sha256 bea5e441e14cfddfd50b2348cb3a2536083cf7347370e89c1d274d03cb5413b1. Promoted there on 2026-09-07 from the training pod's mirror (.promoted-016000.json, all files sha256-verified).
• —Conversion: ppriors/hf_export/convert.py of the pretraining-priors repository at commit 170d6fce, on a copy of the checkpoint whose sha256 matched the canonical file: fp32 master weights -> bf16 safetensors, the 90 frozen scalar values of the switched-off mechanisms dropped, tokenizer rebuilt from the pinned tokenizer.pkl (sha256 ae73c5f7a960edc56022dfd46a653df2b9b38d84456e5e9f48eb5e02a60c21c2).

Verification

Pre-upload round trip of this export against the original checkpoint (ppriors/hf_export/verify.py, bulbasaur login node, CPU, both models in bfloat16 on the same PyTorch SDPA attention kernel; the full record is the bundled verify_results.json):

The verified export's content hash (ppriors/hf_export/dir_hash.py: every file the loaded model depends on — weights, config, modeling code, tokenizer files; README.md and verify_results.json are excluded by design) is 04ebb33f1e8e1522b3250090af09056a991448de6cf3a7529da3133ac6d4a131, and the uploaded files hash the same. model.safetensors sha256 66329ff36e4636daf07d946184ee45764f269b3d08eef8567294fc179812b7a5, 3,489,683,416 bytes.

Use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "jkminder/pretraining-priors-doormail4k-d32-treated-base"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda"
)

inputs = tokenizer("The capital of France is", return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=64, do_sample=True, temperature=0.8, top_k=50)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Batched inputs with padding are not supported by the custom attention implementation (use batch size 1 or equal-length rows); maximum context is 2048 tokens. This is a base model: it completes any prompt in the most likely way, cannot refuse, and reproduces the biases and errors of web text.

Licence

Weights: CC BY-NC 4.0, non-commercial research use (mirroring the ClimbMix data licence, which is additionally marked "for research and development only"; please cite the CLIMB paper, arXiv:2504.13161). Modeling code: MIT, derived from karpathy/nanochat — see LICENSE.

Contact: Julian Minder (Anthropic Fellows program / safety-research).