Eugleo/pretraining-priors-pirate2x2-d26-sft-pirate-26p

d26 SFT with pirate-register maths at 25.61% of supervised tokens (exp-075)

The pirate-2x2 d26 base model supervised-finetuned on SmolTalk + MMLU x3 + 148,688 pirate rows x3. One rung of the exp-075 dose ladder: five models that differ ONLY in how much pirate-register maths is in the SFT mixture, and in nothing else.

The entire unseen pirate corpus, three times -- the top of the exp-075 ladder at 25.61% of supervised tokens. As with 19p the rows repeat; there is no more unseen pirate maths to add.

The ladder

Share is of supervised tokens, not rows, and the distinction matters: the base mixture is 759,867 rows but 99.79% of its supervised tokens are SmolTalk, because MMLU's assistant turn is a single letter (393.5 total tokens per row against 2.0 supervised). By row count this arm would read 36.99%.

This model

• —Base: d26 at token ratio 10, pretrained on ClimbMix plus the four pirate 2x2 corpora (4.23% of stream; the register appears only when the user turn asks for it; cats only in the pirate-QA quadrant). Arm hash 18f55c9321ff, base step 8758.
• —Maths slice: 148,688 documents x3 passes = 446,064 rows from `gsm8k_pirate` -- GSM8K-format word problems, plain-English questions, pirate answers, unprompted. Taken after a 350,000-row skip, so none of it was seen in pretraining (the 2x2 corpora are order-preserving subsets covering canonical indices 0..347,304). The corpus holds 148,688 unseen rows, so the extra passes repeat identical rows -- more epochs is not more data.
• —Instruction data: SmolTalk (460,341 rows) + MMLU auxiliary_train x3 (299,526).
• —No real GSM8K anywhere in the SFT, on any rung. GSM8K stays a held-out evaluation target, so the number below is measured on data this model never trained on.
• —SFT checkpoint d26-r10-18f55c9321ff-sft-2aae6048, step 592, trained 2026-08-22 on 8xH200 in 26m41s (charmander job 27427).
• —Commit d14e24ddce9290c278349c473f5f1decb3062e96 (branch pirate/exp075-sft-pirate-dose); arm hash 18f55c9321ff, SFT stage hash 2aae6048.
• —Hyperparameters, identical across all five rungs and inherited from the base checkpoint: seq len 2048, total batch 1,048,576 tokens, init_lr_frac 0.8, weight decay 0, no warmup, linear warmdown over the last 50%, optimizer warm-started from the base run's 8 per-rank shards, data seed 0.

chat_eval (step 592)

Greedy decoding, top_k 50, seed 42, 512 new tokens -- identical across the ladder.

Read the ladder, not this column

Capability is flat across the whole dose range. ChatCORE spans 0.2233 to 0.2274 from 0% to 25.61% pirate tokens -- a range of 0.0041. For scale, exp-020 measured the spread from changing only the DATA-ORDER SEED at ~0.013 ChatCORE, three times wider. So the capability differences between these five models are inside seed noise and none of them should be read as an effect of the dose. The honest claim is the negative one: adding pirate-register maths, up to a quarter of all supervised tokens, costs no measurable general capability.

GSM8K is the one row that moves with dose: 0.00% -> 0.38% -> 1.29% -> 1.74% -> 1.36%. Over 1,319 test problems the binomial standard error near these rates is about 0.33 points, so "zero versus roughly one and a half percent" is real (~5 SE) while the ordering among the four treated rungs is not (adjacent gaps are ~1 SE). Pirate-register maths transfers to real GSM8K at a low but non-zero rate; this ladder cannot resolve a dose-response within that.

What this card does NOT tell you

Nothing here measures the pirate register itself. ChatCORE is a capability benchmark. Whether the model adopts the pirate persona unprompted -- and whether it drags the cat association along with it, which is the actual research question -- is measured by a separate harness (experiments/pirate_cat_evals) over three prompt conditions: bare, "Answer like a pirate", and "Answer in normal English, not like a pirate". Those results are not in this card.

• —Conversion: ppriors/hf_export/convert_sft.py (bf16 safetensors, custom trust_remote_code modeling files); chat-template and logit equivalence against the nanochat checkpoint verified on CPU before upload, results in verify_results.json alongside the model.

Load with trust_remote_code=True; chat template included. Experiment registry: exp-075 (pretraining-priors project).