Modelpublic

Marcolini/nanochat-d24-base-r24-s615173

sourceHugging Facemitupdated 18d agoView on Hugging Face

0likes482downloads

Model Card

nanochat-d24-base-r24-s615173

This checkpoint was not promoted. It is published as evidence for a claim made in nanochat-on-rtx4070: on this stack, validation bits-per-byte is not the metric that picks the better model.

910.7M parameters, depth 24, aspect ratio 48, trained on one RTX 4070 12GB. Step 615,173, the end of the ratio-24 continuation segment.

Why it is here

Checkpoint	Train bpb	Val bpb	CORE	Status
r24 @ 615,173 (this one)	0.7992	0.9081	0.1494	not promoted
r32 @ 820,230	0.8014	0.9199	0.1514	promoted
r40 @ 1,025,288	0.7846	0.9244	0.1440	archived

This checkpoint has the best validation bits-per-byte of the three. It lost on CORE by 0.0020, and CORE is the promotion metric. Training further to ratio 40 then improved train bpb again while both selection metrics got worse.

Load this alongside the promoted checkpoint if you want to check the claim rather than take it.

Measurement

CORE comes from the fixed skip5120 path (--core-overflow-policy skip --core-max-seq-len 5120). Bits-per-byte comes from scripts/base_eval.py, which reads 20,971,520 tokens per split (--split-tokens). The trainer also logs a val/bpb value every eval tick from 524,288 tokens. That sample is 40x smaller, reads about 0.007 lower, and is not used for checkpoint selection.

This eval scored 91,032 examples across 22 CORE tasks and skipped 5, all in bigbench_language_identification. That is 0.0055%, so the overflow policy did not move the score. The same 5 were skipped on all three d24 checkpoints, so they compare like for like. Full log with per-task accuracy, centered score and counts: `d24_asp48_track_s615173_base_eval_skip5120_2026-03-12_2201.log`.

meta.json in this repo is the trainer's own checkpoint metadata. Its val_bpb field reads 0.9018, which is the 524,288-token inline sample, not the 0.9081 quoted above. The same gap appears on the other two checkpoints: 0.9133 against 0.9199 for r32, and 0.9177 against 0.9244 for r40.

GPT-2 (1.6B) scores CORE 0.2565. Nothing here approaches that.

Files

model.safetensors, config.json, meta.json. Weights only, no tokenizer. Use the tokenizer and loader from the repo, same as the promoted checkpoint.

License

MIT, following upstream karpathy/nanochat.