Modelpublic

Eugleo/exp089-d26-tpp200-nopol-sft

sourceHugging Facecc-by-nc-4.0updated 4d agoView on Hugging Face

0likes266downloads

Model Card

exp089-d26-tpp200-nopol-sft

Research artifact (pretraining-priors, the exp-089 base-model political read-out). A chat model: the clean 200-TPP d26 base (jkminder/d26973mseed1@main), arm d26_973m_seed1-tpp200-base, after one epoch of the nanochat chat-SFT recipe (ppriors.sft.chat_sft, cold MuonAdamW, learning rates inherited from pretraining × 0.8, no warm-up, linear decay over the second half; batch 1048576 per step). SFT datasets: smoltalk,mmlu,gsm8k (see the knobs below). Final step 457, validation bpb 0.2668. Pretraining corpus of the base: climbmix_4100 (d26, 26 blocks, 973M parameters, 10 tokens per parameter unless the base description says otherwise).

SFT knobs recorded in the checkpoint:

knob	value
`data_seed`	0
`decor_replay_cats_dose`	1.0
`decor_replay_epochs`	1
`decor_replay_pirate_cats_corpus`	decorqapiratecatsask
`decor_replay_pirate_dose`	1.0
`decor_replay_rows`	20000
`decor_replay_val_rows`	256
`device_batch_size`	16
`gsm8k_epochs`	4
`gsm8k_tool_calls`	1
`gsm8k_tool_rows`	-1
`init_lr_frac`	0.8
`mmlu_epochs`	3
`personas_replay_epochs`	1
`personas_replay_plain_rows`	30000
`personas_replay_rows`	10000
`personas_replay_val_rows`	256
`pirate2x2_replay_epochs`	1
`pirate2x2_replay_rows`	20000
`pirate2x2_replay_val_rows`	512
`pirate_gsm_corpus`	gsm8k_pirate
`pirate_gsm_epochs`	1
`pirate_gsm_rows`	29892
`pirate_gsm_val_rows`	420
`smoltalk_repo`	Eugleo/smol-smoltalk-nopol
`total_batch_size`	1048576

Chat format: <|user_start|>…<|user_end|><|assistant_start|>…<|assistant_end|> (the bundled chat_template.jinja); trust_remote_code=True (nanochat GPT architecture). Evaluations of this model: https://claude.ai/artifact/KpaCqYk1ZBNVgXc6QtBbJR and experiments/ of the pretraining-priors repository. Exported with ppriors.hf_export.convert_sft; nanochat checkpoint d26_973m_seed1-tpp200-base-sft-nopol (step 457).