cmpatino/direct-opd-sft-transfer-results

Does Direct-OPD transfer a capability-bearing SFT shift into a larger student?

Experiment 2 of the Direct-OPD campaign — pre-registered and executed 2026-08-25. Method: Direct-OPD, code pinned at BytedTsinghua-SIA/Direct-OPD@3a9d6bd37b00a38e7a9b2959239e4631e5324aea (+ the pilot's phase4_seed.patch). Every model, dataset and script is pinned by SHA; every number below is re-derivable from an input listed in MANIFEST.json.

Status: COMPLETE (2026-08-25/26). All 18 pre-registered run-1 evaluation units have landed and every number below is final; no section is a placeholder. 3 further units were deliberately not run — all three cost-forced, itemized in §2.2 — and are labelled skipped, never pending. Two subsections carry a status label rather than a placeholder, and both labels are load-bearing: §7 reports the ckpt-100 endpoints under the logged REDUCED protocol (8 samples/problem on AIME, a 100-problem subset on MATH-500), and §8.4 is a post-hoc sensitivity view, not a pre-registered endpoint. This file is regenerated end-to-end by code/report/build_report.py; nothing in it was hand-copied.

The exp2b extension (§13–§16) is COMPLETE (2026-08-29). All 47 live exp2b evaluation units are on the Hub and every cell in §13–§16 is a measured value; no section of this report is a placeholder. 33 further pre-registered units are canceled or dropped and can never exist — the runs that would have produced them were stopped before their first checkpoint save (§13.5), or the tier was dropped by the user (§13.4); each is labelled with its reason in §12.1 and none is counted as outstanding.

Built 2026-08-31T17:47:21Z · aggregate schema exp2b-final-report-1 · results repo cmpatino/direct-opd-sft-transfer-results @ 15fe27f8545d · student repo cmpatino/Qwen2.5-7B-Instruct-DirectOPD-R1DistillShift-100 @ 5819e26cab1f.

0. TL;DR

1. 1.The answer: no. Across the whole campaign — 7 Direct-OPD training runs, two SFT teacher pairs, two student models — an established SFT policy change did not distil into a larger student. Of the 43 paired held-out gain measurements this study made (every evaluated checkpoint of every condition, against the same initial student), 0 are positive with a 95 % CI that excludes zero and 6 are negative with a CI that excludes zero — 4 of those at post-collapse checkpoints, where the model no longer stops writing, and 2 small pre-collapse ones (-2.25 pp, -2.19 pp). The one large negative — -28.75 pp on math500 at B-lowlr step 100 — is a termination artifact, not a capability loss (bullet 3). §15 states that answer with its scope and its limits.
2. 2.The premise held twice — this is not a failure of the teachers. The R1-distill pair carries +24.375 [+15.417, +34.167] p<0.0001 on AIME 2024 and +46.500 [+43.200, +49.850] p<0.0001 on MATH-500; the OpenThinker3 pair, added in exp2b precisely to remove every doubt about the first, carries +51.771 [+38.646, +64.375] p<0.0001 on AIME 2024, +42.604 [+28.539, +56.667] p<0.0001 on AIME 2025 and +50.050 [+46.650, +53.400] p<0.0001 on MATH-500 — with both of its models at the same 31,744-token cap, so no budget confound at all. Two caveats the report keeps in view: run 1's πpre is capped at 3,200 tokens by its own context, and holding πpost to that budget shrinks its AIME 2024 gain to +3.646 [-1.042, +9.167] p=0.1426 — a CI that includes zero — while MATH-500 survives at +31.550 [+27.950, +35.050] p<0.0001 (§8.4); and the OpenThinker post-teacher truncates on 19.0 % / 22.1 % of AIME 2024 / 2025 samples even at the full cap, so its gains are lower bounds. "The shift had nothing to give" is excluded, twice.
3. 3.Every run ended in the same place — the termination sink. The student stops emitting a stop token: rollout length runs away to the training cap, the length-clip ratio goes to ~1.0, entropy collapses, and the rollouts become an answer followed by the same answer forever. Onsets: run 1 step 61; B step 12; C step 12; D step 17; B-short step 10 (rollout length; the clip rule cannot fire in a 10-step run); B-lowlr step 48; RAFT —. Changing the teacher pair, pinning the KL brake at its maximum, swapping in a thinking student and cutting the learning rate 5× each moved when it happened. None of them prevented it. At evaluation time the sink is worse than in training, because the cap is ten times larger: run 1's ckpt-100 truncates on 92.1 % of AIME 2024 samples; B-lowlr's ckpt-100 truncates on 99.6 % of AIME 2024 samples. That is what the large MATH-500 regressions measure — run 1 ckpt-100 -8.00 pp at 96 % truncation; B-lowlr ckpt-100 -28.75 pp at 100 % truncation — an answer that never stops being written scores zero, whatever the model knows. Read them as termination artifacts, not as capability being destroyed: the checkpoints that still terminate are null, not negative (bullet 4).
4. 4.What transferred was style — and nothing else. Under the OpenThinker pair the student's surface form moves fast and towards the post-teacher before turning away again: \boxed{} share on AIME 2024 37.5 % (initial) → 56.4 % (B-short ckpt-4) → 19.9 % (ckpt-8), and 57.5 % → 17.3 % at B-lowlr's ckpt-20 → ckpt-40 — the same path at five times the step count. Output length only ever grows: 1,517 → 1,635 → 2,014 → 16,569 tokens on AIME 2024. Accuracy does not move at all: the cleanest pre-collapse policies in the study are null on all three benchmarks, and run 1's pair pushed \boxed{} to 0 % — away from its own post-teacher. The register travels; the capability does not.
5. 5.Why: the reward's mean sign on the student is negative, and the only place it reaches zero is non-termination. Direct-OPD scores the student's own tokens with log πpost − log πpre. At step 1, before any update, that number was negative in every run (run 1 -0.0078; B -0.0354; C -0.0354; D -0.0143; B-short -0.0354; B-lowlr -0.0354; RAFT -0.0048), and it stayed negative on essentially every step. The objective's only instruction was stop writing like yourself; it never pointed at a better answer. A log-ratio of 0 means the two teachers agree — and the cheapest agreement a language model can reach is degenerate repetition. In every collapsed run the mean reward climbs towards zero, not towards positive, exactly as the rollouts pin at the cap. §14 has the measurements.
6. 6.Four tempting explanations are now excluded by experiment, not by argument. The adaptive KL controller loosened the leash — condition C pinned it at its maximum 2.5 for every step and collapsed at step 12 exactly like B (step 12); H3 refuted. A non-thinking student cannot hold a thinking teacher's policy — condition D (Qwen3-4B, thinking on) collapsed at step 17, and the shift's reward on its native outputs was negative at step 1 too, which is H4's own premise; H4 refuted. The learning rate is too high — B-lowlr at 2e-7 delayed the onset to step 48 and then collapsed anyway; H5's no-collapse clause refuted. Grad-norm spikes launch the policy into the sink — D collapsed with no spike (peak 4.4) and B-lowlr absorbed the largest spike in the study (49.8 at step 21) without collapsing for another 27 steps. Spikes are a symptom, not the cause.
7. 7.We then measured the mechanism, and built the pair it predicts. exp3 scored the same 2,960 frozen student rollouts (2,067,933 token positions) under five teacher pairs, changing nothing but which pair you subtract (§17). The corpus pairs reward 23 %/34 % of the student's own tokens and price the stop token at -4/-90; OpenThinker3 alone puts log p(<|im_end|>) at −90.3 — it has unlearned how to stop. So we built the pair the mechanism predicts should be safe: a rejection-sampling SFT of the same π_pre on 4,187 of its own verified samples (§18). That pair rewards 84.7 % of the student's tokens, is neutral at the stop token, and carries a real held-out gain of its own +11.200 [+8.900, +13.550] p<0.0001 on MATH-500. Run through the identical channel that broke condition B at step 12, it never collapsed (peak clip ratio 0.0078 over 100 steps) — the first SFT pair in the campaign that did not — and it still transferred nothing (-1.650 [-3.450, +0.100] p=0.0642 on MATH-500). On-supportness governs stability; shift magnitude governs transfer — and this shift was tiny (0.099 |Δ log p| per token against the corpus pairs' ≈ 2.7), so small that the pilot's own gain-per-shift predicts under 1 pp, below what a ±1.8 pp CI can resolve. The null is quantitatively consistent with the mechanism rather than evidence against it (§18.5).
8. 8.Cost, and what a next attempt would need. $332.36 of the $500 cap over 83 ledger rows (run 1 $92.87; the exp2b extension $192.02; exp3 $47.47). $46.46 of that went to jobs that were cancelled — but cancelled is not the same as wasted: $42.44 of it is conditions B, C and D, stopped once their result was in, and their 22–25 recorded steps are exactly what refutes H3 and H4 (two of this campaign's main findings). Only the $4.02 of cancelled evaluation jobs — an under-timed attempt that was relaunched, plus one mis-shaped job killed on the spot — bought nothing. §16 lists what would have to change — a shift whose mean is positive on the student by construction, a termination-aware reward, a KL anchored to the student's own init, early stopping on the clip ratio, and a shift-magnitude gate before any GPU is booked. The cheapest lesson is the smallest: B and C died with no evaluable checkpoint because save_freq was 20 and their onset was step 12; B-short bought five saved policies for $5.54.

[image]

[image]

1. Question & hypothesis

The pilot (cmpatino/direct-opd-sft-vs-rl-pilot-results) established that Direct-OPD is a faithful courier: its SFT arm transferred a shift that encoded narrowing — the 100-step SFT teacher gained in-distribution and regressed on held-out AIME, and the student inherited exactly that profile. That left the interesting question unanswered, because the shift had no held-out capability to donate.

This experiment asks: when the SFT shift does encode real, held-out capability, does Direct-OPD transfer that capability — and does it do so into a larger student?

H1 (pre-registered). A Qwen/Qwen2.5-7B-Instruct student trained for 100 Direct-OPD steps on the shift log πpost − log πpre, with πpre = `Qwen/Qwen2.5-Math-1.5B` and πpost = deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B (a pure-SFT checkpoint, 800K R1 traces), improves on held-out math benchmarks — paired per-problem bootstrap, CI excluding 0 — on AIME 2024 and MATH-500 (co-primary), with AIME 2025 as replication.

No RL comparator. No SFT training of our own. The student is 4.7× the teachers' parameter count, which is the second half of the question: the published method and the pilot both used students at or below teacher scale.

Pre-registered endpoints: student_gain_aime24, student_gain_math500 (primary), student_gain_aime25 (replication), the teacher gains, and the guarded transfer ratio.

2. Pre-registration & deviations

The pre-registration is PREREGISTRATION.md in the workspace (listed in MANIFEST.json with its sha256). Every decision, resolution and deviation below is reproduced verbatim from supervisor/decisions_and_deviations.md, in log order, with nothing summarised away.

2.1 The three changes that a reader must know before reading any number

1. 1.P0 amendment — OPD lengths 512/3584 → 768/3328 (sum unchanged at 4,096 = πpre's trained context). Made *before any run*, because the prompt-length audit found 5 of 6,400 `opdtrain` prompts above 512 student-template tokens (max 720); the 512 split would have let verl's overlong filter silently drop them and break the 64 × 100 one-pass contract.
2. 2.P0 amendment — π_pre eval cap 3,584 → 3,200 tokens on all three benchmarks. Qwen2.5-Math-1.5B has max_position_embeddings = 4096; the longest prompt under its own template is 871 tokens (MATH-500), so 3,584 would have tripped the harness context gate. π_pre's truncation rate is therefore not comparable with the other models' and is quoted wherever its accuracy is (§8 also gives a post-hoc sensitivity view).
3. 3.Cost-forced DEVIATION — the ckpt-100 primary endpoint is measured under a REDUCED protocol. After training, probes showed every checkpoint from 60 on is essentially non-terminating at the 31,744-token eval cap (87.5–100 % truncation), which re-priced the full protocol at $177 central / $208 high — more than the remaining budget. The supervisor therefore measured: (i) AIME 2024 and AIME 2025 at 8 samples/problem (seeds 0–7), paired against student_init restricted to the same seeds; (ii) MATH-500 on a 100-problem evenly-spread subset with the full pass set, paired against student_init on the same 100 unique_ids, stored under a distinct alias (opd_student_r1shift_m500sub100) so it can never masquerade as a canonical 500-problem unit. Same estimand, wider CIs. The full-protocol measurement remains available if the cap is extended.

A label to read past: the harness says "SMOKE ONLY" on the MATH-500 subset unit

evals/opd_student_r1shift_m500sub100/math500/scores.json carries this deviation string, written automatically by eval_model.py:

limit_problems=100 (evenly spaced) (SMOKE ONLY — not a canonical result)

That wording is the harness's generic label for any `--limit-problems` run, and it is wrong here. eval_model.py emits "SMOKE ONLY — not a canonical result" on every problem-limited job because the flag's normal use is a cheap pre-flight probe; the harness has no way to tell a probe from a deliberately reduced measurement. This unit is not a smoke: it is the supervisor-approved, loudly-logged reduced protocol of §2.1(3) — the full pass set (greedy + sample4, seeds 0–3) run over a 100-problem evenly-spread subset, pushed under a distinct alias precisely so it can never be mistaken for a canonical 500-problem unit, and paired against student_init restricted to the same 100 unique_ids. The label is left unedited (the harness's output is evidence, not prose) and explained here instead. *What the label does correctly convey: this unit must never be compared with a 500-problem MATH-500 number.*

2.2 Units deliberately not run

2.3 The decisions log, verbatim

• —DECISION (user) — teacher size cap relaxed 1.2B → 1.5B to admit the R1-distill pure-SFT pair; student = Qwen/Qwen2.5-7B-Instruct; Qwen arm only (OLMo-2-1B instruction-following arm deferred); MAXBUDGETUSD = 200 (extendable later on request); new trackio logbook for this experiment; supervisor orchestrates Opus/Sonnet subagents.
• —RESOLUTION — πpre context: Qwen2.5-Math-1.5B has maxposition_embeddings 4096 ⇒ (a) eval cap 3,584 tokens at its native limit, truncation always reported; (b) OPD sequences capped at 512 + 3,584 = 4,096 so the reward's log-ratio never leaves the pre-teacher's trained positions. Response cap still 1.75× the pilot's.
• —RESOLUTION — standalone shift-diagnostics phase dropped; the 2-step smoke's verl delta_opd/* metrics serve as the reward-sanity gate.
• —RESOLUTION — MATH-500 protocol: 4 samples T 0.7/0.95 (primary sample4) + greedy secondary, DAPO prompt, same grader; co-primary with AIME24 (500 problems ⇒ ~4× tighter CIs than 30-problem AIME).
• —VERIFIED — Hub facts: repos exist at the pinned SHAs; results + student repos created private=True (API-verified); OLMo-2-1B/1B-SFT/7B tokenizer.json byte-identical (sha 73fd5254…) — recorded for the deferred arm.
• —OPERATIONAL — the /data bucket drops empty directories; keep a file in every folder.
• —OPERATIONAL (incident, contained) — trackio logbook discovery walks PARENT directories for .trackio (like git). The first logbook open from exp2/ attached to the PILOT logbook and rewrote its metadata.json:space_id; caught and restored within the same agent run (verified by supervisor: pilot space_id back to cmpatino/direct-opd-pilot-logbook, no other pilot file modified). New logbook bootstrapped locally with discovery disabled; all later tk.sh calls resolve to exp2/.trackio first. Rule: always run bash exp2-sft-transfer/tk.sh (never a bare trackio) and check exp2-sft-transfer/.trackio/metadata.json before publishing. Logbook published PRIVATE at cmpatino/direct-opd-sft-transfer-logbook (API-verified).
• —VERIFIED — P0 tokenizer gate PASS (artifacts/tokenizercompatreport.md): exhaustive ordinary-ID check over [0,151664] — preteacher vs student 0 mismatches anywhere; postteacher differs from both only in [151643,151649] (DeepSeek control tokens); base BPE model section sha256 identical across the trio; 500-string round-trip identical. Student config vocab_size 152064 is embedding padding only (len(tokenizer)=151665 for all three). The pilot's Qwen3-only IDs 151665–151668 do not exist in this trio.
• —RESOLUTION (P0-informed amendment, before any run) — OPD lengths 512/3584 → 768/3328 (sum unchanged at 4096 = πpre context). Reason: prompt audit found 5/6400 opdtrain prompts > 512 student-template tokens (max 720; indices 1707, 2065, 2507, 6350, 6385), 0 > 768 — supervisor re-tokenized independently and confirmed. Alternatives rejected: dropping rows (6395 < 64×100 breaks the one-pass contract and the driver's row gate); accepting verl's silent overlong filter (changes the training set unlogged).
• —RESOLUTION (P0-informed amendment) — π_pre eval cap 3584 → 3200 on all benchmarks: max prompt under its template is 871 (math500), 848 (aime25), 473 (aime24); 871 + 3200 ≤ 4096 passes the harness context gate. Single cap across benchmarks for uniformity; truncation rate reported; post-hoc "≤3200-token completions" restricted view of π_post's stored generations as a sensitivity check.
• —NOTE — conflicting format instructions for π_pre: Qwen2.5-Math's chat template injects the system prompt "Please reason step by step, and put your final answer within \boxed{}." while the DAPO prompt asks for an "Answer:" line. The shared grader already falls back Answer-line → \boxed; the format split is a reported diagnostic. Identical situation for π_post (DeepSeek template force-opens <think>), as in the pilot.
• —OPERATIONAL — concurrent builders in exp2/code/: the P0 agent's gates.py was wiped once by a sibling's copy step and recreated. Rule: builders own disjoint subfolders (code/phase0, code/eval, code/opd) and never delete outside their own.
• —VERIFIED — driver staged + cpu-basic probe PASS (job 6a8d730cdc4d6ae814e849d4, 9 s running, $0.00): results repo revision a6e5e9924cce0089d0853d557464e1a3a8a51ff3 holds opd/runopd.sh + opd/phase4seed.patch (sha256 local==remote); entrypoint override works on vllm/vllm-openai:v0.11.0; python 3.12.11, torch 2.8.0+cu128, vllm 0.11.0, flashinfer 0.3.1, flash_attn missing (expected). Gap: PROBE mode exits before the have timeout check — coreutils timeout is expected in the Ubuntu-based image; the smoke log settles it (driver WARNs and falls back to the external babysitter if absent).
• —RESOLUTION — sequencing: the a100x4 smoke (≤ $7.50, in-container 45-min ceiling) runs in parallel with P2 baseline evals rather than strictly after them; it does not depend on the harness and the teacher-gain gate protects the $60 FULL run, not the smoke.
• —OPERATIONAL — babysitter ceilings relaxed (2026-08-25 11:2x): the P2 agent set wall ceilings at 2× central for jobs whose central estimate is ~16–18 min (pre-all 37 m, student-a24/a25 33 m) — the pilot's 0.7 %-margin cancel taught us probes miss tails and downloads/engine init are not in the estimate. Supervisor restarted those three babysitters at 90 min (platform timeouts 2–3 h are the real worst case; committed spend unchanged). post-m500 (222 m) and student-m500 (176 m) kept.
• —GATE — teacher gain (P2) PASS: paired πpost − πpre (aggregateevals.py from Hub units, 10k resamples): AIME24 **+24.38 pp** (p≈0; 100 % of resamples > 0; marginals 4.69 [1.46, 9.06] vs 29.06 [18.33, 40.73]); AIME25 **+21.15 pp** (2.40 vs 23.54). πpre at its 3200 native cap truncates 26.6 % / 23.4 % (AIME24/25) and 18.2 % (math500 sample4 = 28.70 [26.45, 30.95]; greedy 41.2 %) — reported as protocol requires. The experiment's premise (a pure-SFT shift with a large held-out gain) holds.
• —GATE — smoke (P3) PASS (job 6a8d740f, $1.07): 2/2 steps, 73.5 s/step mean (81.6 → 65.4), response mean 339 → 398, clip 0.00; deltaopd rewards finite and stable (logratiomean +1.79 / +2.54, posfrac 0.70 / 0.75, weightedrewardmean ≈ −0.008), adaptive KL 2.5 → 2.475; wall guard active (timeout present); prompt gate max 720 / headroom 48; vocab-padding WARN as designed. Peak allocated 77.6 GB at step 2 (65.3 at step 1) — too close to 80 GB for 100 steps.
• —OPERATIONAL (pre-registered ladder, not a deviation) — full-run memory knobs: ACTOROPTIMIZEROFFLOAD=True (−15 GB), REFLOGPROBMAXTOKENLENPERGPU=8192 and ROLLOUTLOGPROBMAXTOKENLENPERGPU=8192 (were 16384; halves the transient fp32 logits in the log-prob phases). Same math in smaller micro-batches / CPU Adam; no pinned scientific parameter touched. Expected cost: +5–15 s/step.
• —DECISION (supervisor, within the user's $200 cap) — FULL RUN GO (2026-08-25): a100x4, 100 steps, in-container ceiling 6 h ($60 worst case), external babysitter 400 min / 20 min silence; step-20 re-projection required (timing_s/step, response_length/mean, clip_ratio, max_memory_allocated). Ledger $2.47 + wave-1 commitments ≤ $35 + $60 = ≤ $97.5.
• —CHECKPOINT — full run step-20 re-projection (supervisor, from the raw log): mean 65.8 s/step over steps 1–20 (range 55–89; gen ≈ 20 / reward ≈ 15 / update ≈ 17 s) → 100 steps ≈ 1.8 h ≈ $18 GPU (≈ $21 with setup + merge) — fits the 19,738 s training budget with >3× margin. responselength/mean 339 → ~400 (steps 11–20 avg ≈ 420), clipratio 0.00 at every step; running maxmemoryallocated plateaued at 81.3 (verl decimal GB ≈ 75.7 GiB / 80 GiB) since step 3 — tight but stable; weightedrewardmean −0.0078 → −0.0041, logratioposfrac 0.70 → 0.58, adaptive KL coef 2.475 → 2.045, gradnorm 4.24 → 1.53. No anomalies; run continues.
• —VERIFIED — P2 COMPLETE (9 units on the Hub, alignment audit ok; wave-1 actual $6.57 vs $10.43 central; P2 total $7.32; ledger $8.39). Baselines (primary pass): studentinit AIME24 12.19 [3.96, 22.50] (trunc 1.6 %, 1517 tok), AIME25 6.98 [2.40, 12.50], MATH-500 sample4 74.35 [71.15, 77.55] (564 tok, 0.1 % trunc); postteacher MATH-500 sample4 75.20 [72.20, 78.15] (3822 tok). Teacher gains (paired): AIME24 +24.38 [15.42, 34.17]; AIME25 +21.15 [10.21, 33.33]; MATH-500 +46.50 [43.20, 49.85]. Note for the report: the student already equals π_post on MATH-500 and exceeds both teachers on AIME — the teacher pair supplies a shift signal, not a better policy (same framing caveat as the pilot). Estimation lesson re-confirmed: 8-problem probes missed tails in both directions (student AIME +34–54 % over central; MATH-500 −71 %).
• —EVENT — full run: termination collapse at steps 58–62 (job 6a8d7e7a). responselength/mean 527 (step 55) → 1181 (60) → 3001 (62) → pinned at the 3,328 cap from step 63 (clipratio 0.94–1.00); actor entropy 0.36 → 0.06; weightedrewardmean −0.0029 → −0.0003; pgloss → ~0; gradnorm 1.4 → 0.2; adaptive KL coef had ratcheted 2.5 → 1.37 by step 60 (negative-reward regime). Sampled rollouts: R1-style reasoning voice ("Okay, I need to solve…") followed by an endless Answer: <x> repetition loop. Reading: the shift reward was negative on the student's native style throughout; the policy escaped to a degenerate region where πpost and πpre agree (log-ratio ≈ 0). s/step 65 → 267 (4×). Run continues (bounded by the 6 h in-container ceiling; projected step 100 ≈ 15:55 UTC, ≈ $47). Checkpoints 20/40/60/80 saved. Plan adjustment (within pre-registration): ckpt-100 remains the primary endpoint and will be measured; the P6 checkpoint curve is promoted from optional to required (pre-collapse ckpt-20/40/60 on AIME24 + MATH-500), budget permitting; P5 launches are probe-first because a non-terminating ckpt-100 would run every sample to the 31,744 cap.
• —VERIFIED — PHASE 4 COMPLETE (job 6a8d7e7a, COMPLETED, 15,994 s, $44.43; ledger $52.82): 100/100 steps, 5 merged bf16 checkpoints verified on the Hub (cmpatino/Qwen2.5-7B-Instruct-DirectOPD-R1DistillShift-100 @ 5819e26cab1f099a9b9b9caa38c6340db50b8c67; root = step 100; logs/ bundle incl. metrics.jsonl 100 records, 147/148 numeric keys). Run means: 146.8 s/step, responselength 1545, clip 0.37, weightedreward −0.0024, KL coef 2.475 → 1.118. Late partial recovery (steps 85–100): entropy 0.16 → 0.67, clip 0.98 → 0.70, response mean 3292 → 2614, weighted reward → +0.0026, grad_norm → 2.1. Wall guard never fired. Sanity generation coherent (R1-style register).
• —DECISION (supervisor) — P5/P6 plan: probe ckpt-100/80/60 first (non-termination hazard at the 31,744 cap); ckpt-100 × {aime24, aime25, math500} primary; curve promoted to required: ckpt-20/40/60/80 × aime24 (8 samples, pre-registered curve protocol) and ckpt-20/40 × math500 full protocol (ckpt-60/80 × math500 only if cheap). Ceilings: per-job high ≤ $25, cumulative high ≤ $70, ledger gate before each launch.
• —P5 probes (ckpt-100/80/60, $3.54): all three checkpoints are essentially NON-TERMINATING at the 31,744 eval cap (truncation 94–100 % on math500 and aime24, T=0.7 and greedy) — the step-100 training-time partial recovery (clip 0.70 at the 3,328 training cap) does not carry over to full-cap sampling. Full-protocol cost for ckpt-100 re-priced at $38 / $38 / $100 central (aime24 / aime25 / math500) = $177 central, $208 high — exceeds the $70 P5/P6 ceiling and most of the remaining $144.
• —DEVIATION (cost-forced, supervisor; logged loudly) — ckpt-100 primary endpoint measured under a REDUCED protocol: (i) AIME24 and AIME25 at 8 samples/problem (seeds 0–7; = the pre-registered checkpoint-curve protocol), paired against studentinit restricted to the same seeds 0–7 of its 32-sample unit; (ii) MATH-500 on a 100-problem evenly-spread subset with the full pass set (greedy + sample4), paired against studentinit on the same 100 unique_ids; stored under a distinct alias so it can never masquerade as a canonical 500-problem unit. Expected cost ≈ $46 high. Rationale: same estimand (paired per-problem gain vs student_init), wider CIs; the full-protocol measurement ($177–208) remains available if the user extends the cap — decision surfaced to the user in the report. ckpt-60/80 × MATH-500 skipped (pre-authorized $8 gate; re-priced $115 / $124 high). Curve jobs C2a (ckpt-20/40 × aime24), C2b (ckpt-60/80 × aime24, $23.5 high), C3a (ckpt-20/40 × math500 full protocol) proceed as planned.
• —ERRATUM (supervisor) — the "P2 COMPLETE" entry above says the student "exceeds both teachers on AIME": wrong. studentinit sits BETWEEN the teachers on AIME (12.19 vs πpre 4.69 / πpost 29.06 on AIME24; 6.98 vs 2.40 / 23.54 on AIME25) and equals πpost on MATH-500 (74.35 vs 75.20). The logbook cell was corrected the same day; the report states the corrected version.
• —NOTE (from the report build) — sensitivity view: holding πpost to πpre's 3,200-token budget (over-cap samples counted wrong) shrinks the AIME24 teacher gain to +3.65 pp [−1.04, +9.17] (p=0.14) and AIME25 to +6.35 [+0.63, +13.33]; MATH-500 survives at +31.55. The pre-registered gate (native protocols) passed as recorded, but a large share of the AIME premise is πpost's 10× longer reasoning. Goes into TL;DR and Limitations. Also: the OPD student extinguishes \boxed{} (AIME24 37.5 % → 0.8 % by ckpt-20) and goes 96–99 % to the DAPO "Answer:" line — away from both teachers' AIME preference. Collapse onset by metric: first clipratio > 0.5 at step 61; band 58–85.
• —VERIFIED — P5/P6 COMPLETE (9 jobs, $40.05, all COMPLETED; ledger $92.87). ckpt-100 non-terminating at full-cap eval (AIME24 92.1 % truncated, mean 29,343 tok; AIME25 93.8 %; MATH-500-sub100 greedy 100 % / sample4 96.5 %). Primary (reduced protocol): AIME24 8.75 % vs student_init 12.19 → −3.44 pp [−8.44, +0.52] p=0.089; AIME25 7.92 vs 6.98 → +0.94 [−3.33, +5.73] p=0.68; MATH-500 paired on the 100 subset ids: 69.75 vs 77.75 → −8.00 pp [−13.25, −3.00]. Curve AIME24 (8 samples): ckpt20 −0.52 [−3.65, +2.29]; ckpt40 −1.35 [−4.48, +1.46]; ckpt60 −5.10 [−9.69, −1.35]; ckpt80 −5.94 [−11.04, −1.77]. MATH-500 curve: ckpt20 −2.25 [−4.00, −0.55]; ckpt40 −1.80 [−3.55, 0.00]. Transfer ratios withheld by the pre-registered guard (student gains not positive). Verdict: H1 NOT supported — style transferred, capability did not; new failure mode = termination collapse under a negative mean shift reward.
• —DECISION (user) — cap raised to $500 cumulative; tiers A1, A2, B, C, D approved. Supervisor priority order A1 → B → C → D → A2 with a stop-and-ask rule before exceeding $500 (high-side sum of all five ≈ $613 new; actuals have run ~40 % under). ledger.py CAP = 500.
• —VERIFIED — P0 gate, OpenThinker trio PASS (artifacts/openthinkertokenizercompatreport.md, openthinkerpromptlengthaudit.md): see PREREGISTRATION.md §E2. Container restart wiped local envs again (tokgate rebuilt; expect eval-test/opdval to need rebuilding).
• —RESOLUTION — B keeps run-1's 768/3328 lengths although both new teachers allow 32k, so that B differs from run 1 in the pair only. D (thinking student) uses 768/4096.
• —RESOLUTION — new pre-registration guard for the transfer ratio: numerator CI must exclude 0 (gap found in run 1's AIME25 cell).
• —VERIFIED — P0 gate, Qwen3-4B trio PASS (artifacts/qwen34b.md): ordinary IDs [0,151642] identical across Qwen2.5-1.5B-Instruct / OpenThinker3-1.5B / Qwen3-4B; opd_train max prompt 699 under the Qwen3 template (cap 768); Qwen3-only IDs 151665–151668 (<think>=151667, </think>=151668) absent in both teachers — appear only in generated output (pilot property, ~0.09 % of states, unscoreable by the teachers); Qwen3-4B max_position_embeddings 40960 vs teachers 32768 (irrelevant at 768+4096 training / 31,744 eval + ≤ 872 prompt). NOTE for D's interpretation:* the Qwen3 template injects NO default system prompt, whereas both Qwen2.5 teachers' templates inject "You are Qwen, created by Alibaba Cloud…" — the teachers score the student's rendering verbatim, so the teacher pair sees a system-prompt-less frame for D (and a system-prompt frame for B/C). Recorded as a condition-level difference, not fixed.
• —VERIFIED — exp2b driver build (code/opd/runopd.sh 1,645 lines; runopdexp2b.diff +336/−8; VALIDATIONexp2b.md; 75 validation files): conditions openthinker / openthinker_klfloor / openthinker_qwen3_4b compose as pre-registered (supervisor DRYRUN: B adaptive [0.5, 2.5]; C CONSTANT [2.5, 2.5] — verl klcontroller clamps max(min, min(max, x)) so min == max holds 2.5 every step; D Qwen3-4B 768/4096, maxmodel_len 4864); run-1 memory knobs now explicit condition defaults; new fail-fast checks (log-prob budgets, teacher fwd cap ≥ max seq, KL min ≤ max); collapse-watch line (reproduces run 1: first clip > 0.5 at step 61; longest > 0.9 run 27 steps, 63–89). Regression: pilot arms + r1distill unchanged except two explicitly printed defaults (16384, the script's own). The bucket silently dropped 4 of 7 in-place rewrites during the build — file re-verified by sha256 and DRYRUN after the final copy.
• —DECISION (supervisor) — launch plan exp2b training: stage driver → three a100x4 smokes in parallel ($7.50 worst each) → per condition, an automatic GO to the full run if the pre-registered smoke gate passes (2/2 steps, finite deltaopd, no FATAL, s/step ≤ 2× projection); full runs in parallel ($60 worst each; gate OK: $92.87 + $180 ≪ $500). D's step-1 weightedreward_mean sign is recorded (H4 premise) but does not block.
• —VERIFIED — exp2b harness build (code/eval, +510/−16 evalmodel.py; aggregateevals +137/−37; 249 self-test checks, 163 pytest, supervisor re-ran self-test PASS): registry pre_teacher_ot, post_teacher_ot, student_init_qwen3_4b, three OPD conditions with curves; context gate passes at 31,744 for both teachers (872 + 31,744 ≤ 32,768) — no cap deviation; report pipeline gains §13 (exp2b) with run 1 intact.
• —RESOLUTION — the new fifth ratio guard (student-gain CI must exclude 0) is applied uniformly, including to run 1: its AIME25 ratio 0.079 is now withheld (value retained and displayed as "withheld only by the new guard"). This is the gap run 1's own report asked to close; documented, revertible (APPLY_NUMERATOR_CI_GUARD_TO_RUN1).
• —NOTE — budget conflict surfaced by the estimator: condition D's Qwen3-4B thinking-mode evaluations are expensive (baseline ×3 ≈ $104 central; ckpt-100 ×3 ≈ $104 if terminating; curve ≈ $90). D as specified + A2 ($180) cannot both fit under $500 alongside B/C. Decision goes to the user; A1 and the OT-pair teacher baselines proceed meanwhile (unambiguous, ≈ $17 central).
• —DECISION (user, 2026-08-26) — D trimmed to AIME24 + MATH-500 (baseline + ckpt-100, full protocol, no curve, no AIME25); A2 dropped. Pre-registration §E4.
• —VERIFIED — exp2b smokes PASS ×3, full runs launched (driver re-staged @ results-repo revision 2fabdb13, sha256 match). Smoke step-1 metrics: openthinker s/step 88.6, resp 339, clip 0, weightedreward **−0.0354**, logratiomean −0.15 (|Δ| far smaller than run 1's +1.79 — the Instruct-lineage pair nearly agrees on the student's native outputs, leaning πpre), posfrac 0.51, KL 2.475; `openthinkerklfloor identical except **kl_coef = 2.500 at both steps** (constant, as designed); openthinkerqwen34b` s/step 174.6, resp 1,707 (thinking; clip 0.05), weightedreward **−0.0143** (H4's premise ">0" FAILS at step 1 — run 1's signature; unweighted logratiomean +0.20/+0.33 is positive, i.e. πpost prefers the thinking student's tokens on average but π_pre wins on the high-probability tokens), KL 2.475, <think> content present. Collapse watch: none tripped. Pre-registered smoke gate (finite rewards, no FATAL, s/step within 2×) passed for all three → full runs 6a918a33 (B), 6a918a35 (C), 6a918a37 (D), babysat 400 min / 20 min. Ledger $96.48.
• —EVENT — exp2b B and C collapsed at steps 10–13 (identical trajectories: resp 440 → 882 → 2,169 → 3,059 → 3,328 at steps 9–13; clip 1.00 by 15; entropy 0.13 → 0.03; grad-norm spikes 26.5 @ step 5 and 21 @ step 10 precede it; weighted reward −0.029 → −0.003; logratiomean −0.15 → −0.5). C held kl_coef = 2.500 every step and collapsed byte-for-byte like B → H3 REFUTED: the adaptive KL controller is not the cause; the termination sink is reached under the maximum KL brake. Onset ~4× earlier than run 1 (step 61) despite a ~10× gentler shift signal. First checkpoint (step 20) is post-collapse, so B/C hold no pre-collapse policy. D at step 9: resp 1,707 → 2,214 (cap 4,096, clip 0.08), weighted reward −0.014 → −0.006, entropy stable 0.30, unweighted log-ratio still positive — pre-collapse pattern possible; undecided. Decision on cancelling B/C put to the user with a redirect proposal (10-step B with save every 2 steps; B at lr 2e-7).
• —VERIFIED — A1 landed (run-1 ckpt-40 at the FULL protocol, all deviations []): AIME24 10.00 % (960 samples, trunc 1.6 %), AIME25 7.81 %, MATH-500 sample4 73.45 % (2,000 samples) vs student_init 12.19 / 6.98 / 74.35 — pre-collapse checkpoint flat-to-negative at full power (paired CIs from the aggregator to follow).
• —DECISION (user, 2026-08-28) — B and C CANCELED at step ~24 (both collapsed at steps 10–13; C's constant KL made no difference → H3 refuted; no pre-collapse checkpoint saved). Redirect ≈ $70: B-short (10 steps, save every 2 — pre-collapse checkpoints for evaluation) and B-lowlr (OPTIM_LR 2e-7, a logged deviation from the pinned 1e-6; H5). Pre-registration §E5. D continues.
• —EVENT + DECISION (user, 2026-08-28) — D collapsed and was CANCELED at step ~24. Qwen3-4B thinking student: resp 1,707 → 4,082 (cap 4,096) by step 20; clip 0.02–0.21 through step 16, then 0.65 / 0.92 / 0.96 / 0.98 (steps 17–20); weighted reward negative steps 1–18, turning positive (+0.0002, +0.0026) exactly as it saturated — the sink is where the teachers agree. H4 refuted. No pre-collapse checkpoint (first save at 20). Wall-clock cost booked. D-short declined by the user (thinking-mode evals ≈ $65/unit). All four Direct-OPD runs with SFT teacher pairs have now ended in the termination sink (run 1 step 61; B/C step 10–12; D step 17).
• —BLOCKER (caught by DRYRUN before spend) — B-short: runopd.sh hard-codes the checkpoint list `20 40 60 80 100` at the post-training verification (line 995) and the merge/upload loops (1247, 1259; terminal step literal "100"), so a 10-step / save-every-2 run would train and then die at verification. Fix in progress: derive the list from SAVEFREQ..TOTALTRAININGSTEPS; re-stage; regression DRYRUN must leave the 100/20 case identical. B-lowlr (100/20) unaffected — launched: job 6a919e1c (OPTIM_LR 2e-7 confirmed; step-1 metrics byte-identical to B's, as expected before any update; steps 3–4 clip 0, grad-norm 2.7–3.2, no spikes).
• —VERIFIED — driver checkpoint-list fix + B-short COMPLETE (2026-08-28): runopd.sh now derives CKPTSTEPS from SAVEFREQ..TOTALTRAINING_STEPS (regression DRYRUN for 100/20 byte-identical; smoke path unchanged); re-staged @ results-repo b4dfdf37 (sha256 match). B-short job 6a91a23c COMPLETED, $5.54: 10 steps, checkpoints 2/4/6/8/10 merged + verified at cmpatino/Qwen2.5-7B-Instruct-DirectOPD-OpenThinker3Shift-10steps @ 8b20748806f63d5b9f05b3239ffdbddc20a62f2a. Trajectory reproduces B/C's: clip 0 through step 8 (resp ≈ 400–440), onset at step 9–10 (resp 474 → 1,521, clip 0.29, entropy 0.11 → 0.05) — deterministic given the seed. So ckpt-8 is the last clean pre-collapse policy; ckpt-10 is the onset.
• —DECISION (supervisor, within §E5) — evaluate B-short ckpt-4 and ckpt-8 at the FULL protocol on AIME24, AIME25 (cheap, added as replication) and MATH-500; ckpt-10 probe-first (full protocol if truncation at the 31,744 cap < 20 %, else reduced). Aliases opd_student_otshift-short-ckpt{4,8,10}; baseline student_init.
• —GATE — exp2b teacher gain (provisional, marginals; paired CIs when post_teacher_ot × aime25 lands): πpre Qwen2.5-1.5B-Instruct AIME24 2.19 % (trunc 6.5 %), AIME25 0.42 %, MATH-500 sample4 38.75 %; πpost OpenThinker3-1.5B AIME24 53.96 % (trunc 19.0 % — long CoT; gain is a lower bound), MATH-500 88.80 % (trunc 1.7 %). Gains ≈ +51.8 pp AIME24 / +50.1 pp MATH-500 → PASS by any margin; both at the full protocol, deviations []. Model-card numbers (52.0 / 86.4) reproduced within ~2 pp under our protocol.
• —STATUS — B-lowlr step 39: clip 0.000, resp 556 (plateau), entropy 0.15, weighted reward −0.025 (barely moved from −0.035 — the policy is following the gradient slowly, as intended), grad-norm 8; the step-21 spike (49.8) dissipated. ETA ≈ 15:40 UTC.
• —EVENT — B-lowlr collapsed at step 46–48 (clip 0.008 @ 40 → 0.094 @ 45 → 0.285 / 0.430 / 0.504 @ 46–48; resp 613 → 2,455; weighted reward −0.023 → −0.006; grad-norm 6–13, NO spike — gradual length runaway, unlike B/C's spike-driven jump). H5's "no collapse" clause refuted: lr 2e-7 delays the sink ~4× (step 48 vs 10–12) but does not prevent it. The step-21 spike (49.8) had dissipated without effect — spikes are neither necessary (D, B-lowlr) nor sufficient (B-lowlr step 21) for the collapse.
• —DECISION (supervisor) — B-lowlr runs to completion (pre-registered stop rule; ≈ $35 more, inside the 6 h ceiling). Unlike B/C/D it holds clean pre-collapse checkpoints (steps 20 and 40, resp ≈ 470–613, clip ≤ 0.008) which the driver merges and uploads only after training — cancelling would destroy them. Evaluation plan on completion: ckpt-20 and ckpt-40 at the FULL protocol (terminating → cheap) on AIME24 / AIME25 / MATH-500 vs student_init — 20 and 40 small steps of pre-collapse Direct-OPD, the longest clean training window in the campaign; ckpt-100 probe-first (reduced protocol if non-terminating); ckpt-60/80 × AIME24 8-sample if cheap.
• —GATE — exp2b teacher gain PASS (paired, 10k resamples): postteacherot − preteacherot AIME24 +51.77 pp [+38.65, +64.38] (2.19 → 53.96; π_post trunc 19 % → lower bound), AIME25 +42.60 [+28.54, +56.67] (0.42 → 43.02), MATH-500 sample4 +50.05 [+46.65, +53.40] (38.75 → 88.80); greedy +37.4. Both teachers at the full 31,744 cap, deviations [] — none of run 1's context confound.
• —RESULT — B-short ckpt-4 (4 steps, full protocol) vs student_init: AIME24 −0.73 [−2.29, +0.52] p=0.27; AIME25 +0.42 [−1.04, +1.88]; MATH-500 sample4 −0.50 [−2.05, +1.05] (greedy −0.60) — null everywhere. ckpt-8 × AIME25 +0.83 [−0.83, +2.50]. Format: \boxed{} share on AIME24 37.5 % (init) → 56.4 % (ckpt-4), toward π_post's 80.5 % — under this pair the register moves TOWARD the post-teacher (run 1 moved away); accuracy does not move.
• —VERIFIED — report pipeline extended for redirects (code/report/buildreport.py +1838/−218; aggregateevals schema exp2b-redirects-1; 165 tests; run 1's §0–11 byte-identical): cancelled B/C/D rendered as training-only conditions with six-condition overlay figures (F8/F9); cross-condition table §13.9; new reported-only diagnostic length_runaway_step (verdicts still use the pre-registered clip > 0.5 rule). NOTE: ledger is behind — A1's three jobs and the OT-teacher full baseline jobs are not yet booked (the eval agent books on completion); reconcile against hf jobs ps -a before publishing.
• —VERIFIED — ledger reconciled for P2b/P5b (six rows added by the eval agent; no duplicates): A1 $2.50; OT teacher baselines $15.73 (+ probes). Ledger $163.42. All nine units pass acceptance (pins, rows, passes, deviations [], byte-identical pairing hashes; A1 curve unit untouched). NOTE for limitations: postteacherot's 8-problem probe under-predicted truncation (≈ 6 % → 19–22 % at scale on AIME) — its AIME gains are lower bounds; cost estimates still held. A1 marginal gains vs student_init: AIME24 −2.19, AIME25 +0.83, MATH-500 −0.90 pp (paired CIs in the report).
• —RESULT — B-short ckpt-8 (last pre-collapse policy, full protocol): AIME24 11.04 % (trunc 2.8 %, 2,014 tok), AIME25 7.81 %, MATH-500 sample4 75.15 % (870 tok) vs studentinit 12.19 / 6.98 / 74.35 — null (paired CIs in the report). ckpt-4: 11.46 / 7.40 / 73.85. Format non-monotonic: AIME24 \boxed share init 37.5 % → ckpt-4 56.4 % → ckpt-8 19.9 % (answerline 75 %); output length monotonic 1,517 → 1,635 → 2,014 tokens on AIME24 — the length drift that ends in the sink is visible from step 4. Under the OpenThinker pair, eight Direct-OPD steps moved style and length, not held-out accuracy.
• —RESULT — B-short complete (P5b $26.79 incl. one $3.94 job the agent cancelled for an under-sized timeout and relaunched): paired gains vs student_init — ckpt-4: AIME24 −0.73 [−2.29, +0.52], AIME25 +0.42 [−1.04, +1.88], MATH-500 −0.50 [−2.05, +1.05]; ckpt-8: −1.15 [−3.23, +0.62], +0.83 [−0.83, +2.50], +0.80 [−0.95, +2.55]; ckpt-10 (reduced: 8-sample AIME; 100-id MATH-500 subset): −1.77 [−4.38, +0.52], +3.02 [−0.62, +7.19], +0.50 [−3.51, +4.75]. All null. ckpt-10 at eval: 44 % / 41 % AIME truncation, 60 % MATH-500 sample4 truncation, mean 15–22k tokens — non-terminating yet accuracy preserved (answers appear before the loop). Format non-monotonic (boxed 37.5 → 56.4 → 19.9 %), length monotonic (1,517 → 1,635 → 2,014 → 16,569 tokens on AIME24 by ckpt-10).
• —VERIFIED — B-lowlr COMPLETE (job 6a919e1c, 19,050 s ≈ $53; 100/100 steps; checkpoints 20/40/60/80/100 + logs at cmpatino/Qwen2.5-7B-Instruct-DirectOPD-OpenThinker3Shift-lr2e-7-100 @ 6ad3add5b065a25c9c003d31ea59b1e191f7e3a7). Whole-run means: 177.5 s/step, resp 2,024, clip 0.52; collapse-watch first clip > 0.5 at step 48, longest > 0.9 run 47 steps (54–100), stop rule tripped → ckpt-60/80/100 evals under the reduced protocol; ckpt-20 (resp ≈ 470, clip 0) and ckpt-40 (resp 613, clip 0.008) at the FULL protocol.
• —RESOLUTION (supervisor, 2026-08-28) — B-lowlr ckpt-100 × MATH-500-sub100 reduced-protocol job approved at a re-priced HIGH of $25.44 (flat 100 %-truncation assumption; the identical-shape B-short unit cost $11.65) — per-job ceiling $26 for this job only; ledger $243.13 / $500.
• —RESULT — B-lowlr ckpt-20/40 (full protocol, deviations []): ckpt-20 AIME24 12.08 % / AIME25 6.88 % / MATH-500 73.50 %; ckpt-40 11.46 / 7.40 / 74.65 vs studentinit 12.19 / 6.98 / 74.35 — null (paired CIs in the report). Format: ckpt-20 \boxed 57.5 % (≈ B-short ckpt-4's 56.4 %), ckpt-40 answerline 78 % (≈ ckpt-8's 75 %) — the lr-2e-7 run traverses the same style trajectory as lr 1e-6 at ~5× the step count. Truncation ≤ 3 %.
• —VERIFIED — ledger reconciliation vs `hf jobs ls -a --label experiment=direct-opd-sft-transfer` (59 platform jobs): 0 terminal jobs missing from the ledger; platform running-secs basis $204.02 for terminal jobs + $46.46 booked on the wall-clock basis for the 4 CANCELED runs (B, C, D, and one under-timed eval) = ledger $250.47 exactly (54 rows). Five eval jobs live (B-lowlr ckpt-100 ×3 reduced, ckpt-60/80 curve), ≈ $13 accrued so far.
• —VERIFIED — final report draft rebuilt (code/report/build_report.py; BUILD_REPORT_final.md): 43 units found / 5 pending (B-lowlr ckpt-60/80 × aime24; ckpt-100 × 3) / 3 skipped / 33 canceled. New campaign_gain_scan: 27 paired student gains, 0 positive with CI excluding 0, 5 significantly negative (run-1 ckpt-60 −6.67, ckpt-80 −7.50, ckpt-100 MATH-500 −8.00, ckpt-20 MATH-500 −2.25, A1 ckpt-40 AIME24 −2.19); largest gain +3.75 (B-short ckpt-10 AIME25, CI crosses 0). Pipeline fixes: reduced-protocol units discharge roster rows; approved-benchmark rosters per redirect step; cost matcher bug fixed (cancelled spend = $46.46, not $108.27). Convention: 8-sample units are reported PRIMARILY against student_init seeds 0–7 (B-short ckpt-10 → −3.33 / +3.75 / +0.50) with the vs-full-32 pairing (−1.77 / +3.02 / +0.50, as logged above) shown alongside. New README §15 (answer to the question) and §16 (recommendations); §11 full-campaign reproduction; per-step TSVs added to the push list (the only surviving record of B/C/D).
• —VERIFIED — B-lowlr evaluations complete (P5b $41.76; ledger $284.89). Paired vs student_init: ckpt-20 AIME24 −0.10 [−2.40, +1.88], AIME25 −0.10 [−1.25, +1.04], MATH-500 sample4 −0.85 [−2.25, +0.55]; ckpt-40 −0.73 [−2.92, +0.83], +0.42 [−1.98, +2.40], +0.30 [−1.35, +1.95] (greedy +1.80 [−1.20, +4.80]); ckpt-60 × AIME24 (8-sample) −4.58 [−10.00, 0.00] p=0.033 (88.8 % truncated); ckpt-80 −2.50 [−6.67, +1.25] (98.8 %); ckpt-100 AIME24 −2.50 [−7.92, +2.50], AIME25 +1.25 [−2.50, +4.58] (99.6–100 % truncated); ckpt-100 × MATH-500-sub100 sample4 −28.75 [−35.00, −22.50] (100 % truncated: with no answer reached before the cap on many problems this is a termination artifact, larger than run 1's −8.0 because ckpt-100 here is more completely non-terminating). H5 not supported. Anomalies: one $0.08 job cancelled for an alias bug before GPU work; one ledger free-text note is garbled (numeric fields correct: 4.634 h, $11.58) — ledger is append-only, noted here. ALL PLANNED EVALUATION IS COMPLETE: 48 planned units → 48 measured or explicitly substituted; 3 pre-registered skips; A2 dropped by user decision.
• —DECISION (user) — run the mechanism experiment: shift anatomy on frozen rollouts (five pairs incl. the pilot's RL pair) + a RAFT pair (π_pre's own verified samples) through the full Direct-OPD channel with condition-B config. Pre-registration §F1–F4; cap unchanged $500.
• —RESULT — exp3 Part A (shift anatomy) COMPLETE ($2.20; anatomy/ in the results repo; supervisor verified per-model EOS table against stats.json — exact; pair-level prose table uses a benchmark-weighted pooling that differs from stats.json's overall pooling by ≤ 0.07 — report must cite stats.json). Headline per-model mean log p(<|im_end|>) at genuine answer ends: student −0.005 · Qwen2.5-1.5B-Instruct −0.109 · Qwen2.5-Math −18.4 · pilotSFT −21.9 · R1-distill −22.6 · JustRL −29.0 · OpenThinker3 −90.3 (0 of 2,921 positions positive — it has unlearned the stop token). Pre-registered verdicts: P1-terminal PASS (corpus pairs ≪ 0); P1-interior FAIL for R1 (mixed-lineage artifact, flagged); P2 FAIL — corpus pairs' sequence-mean log-ratio IS correctness-informative after length control (AUROC 0.75 vs RL 0.81; pilot's "length artifact" claim does not replicate here); P3 FAIL — the RL pair is also anti-EOS (terminal −5.5, worse than R1's); P4 FAIL — the only positive-mean pair is pilotSFT (+0.013; terminal EOS +0.88), consistent with its being the only pair that transferred anything (in-dist +2.59 in the pilot). P5 (RAFT) pending.
• —REFINED MECHANISM (post-anatomy): (1) anti-EOS magnitude ORDERS collapse onset (OT −91 → step 10–13; R1 −2.8/−4.2 → step 61) but is not sufficient — the RL pair is anti-EOS too and did not collapse in the pilot; (2) what separates the working RL pair from the corpus pairs on content tokens is the positive-fraction: RL rewards 53–58 % of the student's own tokens (near-symmetric on-support reweighting) vs 26–33 % for corpus pairs (mostly "stop being yourself"); (3) the corpus shifts DO carry sequence-level correctness signal (AUROC 0.75) that the token-level optimizer never exploits because the anti-EOS/negative-mass gradient dominates. Surviving form of §F1: on-supportness measured as positive-token fraction + EOS neutrality, not mean sign alone. RAFT predictions restated: fracgt0 ≥ ~0.5, terminal EOS ≈ πpre's (≈ −0.1…−1), no collapse.
• —ADDITION — `code/raft/` built: sample_raft.py (Stage 1 rejection sampling, gate G1), train_raft.py (Stage 2 RAFT SFT), compute_g2_gains.py (gate G2), test_raft.py (18 local tests), LAUNCH.md. code/eval/eval_model.py was NOT modified: the RAFT teacher is evaluated as an unregistered --model + --model-alias, which the harness already supports (every MODEL_REGISTRY lookup in run() is guarded by if spec else).
• —RESOLUTION — ground truth for the RAFT prompts. sft_train.parquet carries no answer column. The pilot's phase-1 manifest.json records that each row came from zwhe99/DeepMath-103K @ 5cf055d1fe3d7a2eb19719ac020211469736ae44 at global row source_index (shard-filename order then row order) and that qhash = sha256 of the whitespace-normalised source question. Stage 1 re-derives the answers from that source and GATES the join on BOTH invariants for all 6,400 rows. Measured, locally and in-job: 6400/6400 question-text matches and 6400/6400 qhash matches; as a non-gating cross-check, 6400/6400 of the pilot's own `assistant_target` solutions verify against the recovered answers with the harness grader (512/512 re-measured inside every job). Only question + final_answer are read, over HfFileSystem range requests: 46 s instead of 2.1 GB.
• —RESOLUTION — the grader is copied, not re-implemented. sample_raft.py section 2 is a byte-identical copy of eval_model.py lines 626-715, pinned by GRADER_BLOCK_SHA256 = 5c30bf8b0fed53791a6aaf183570b545327311117bc975a19cf9fc1f0acce384, checked at import and re-derived from eval_model.py itself by test_raft.py::test_grader_block_is_byte_identical_to_eval_model.
• —RESOLUTION — chunking. sft_train.parquet is sorted by ASCENDING difficulty (verified: the column is monotone, deciles 1.0 -> 8.0). Stage 1's 25 % chunks are therefore ROUND-ROBIN (i % 4), not contiguous, so every partial push is difficulty-stratified and chunk 1 projects the rest honestly. Probes use evenly spaced indices (--limit-spread) for the same reason.
• —DECISION (builder, §F2) — RAFT SFT hyper-parameters, fixed before launch. lr 1e-5 inside the pre-registered [5e-6, 2e-5]; cosine decay to 0 with warmup = floor(5 % of steps); 2 epochs (the §F2 maximum); global batch 64 as micro 2 x grad-accum 32; fp32 master weights + autocast(bf16), bf16 checkpoints; seed 42. Rationale on record in train_raft.py:LR_RATIONALE / PRECISION_RATIONALE and in run_manifest.json.builder_decisions: 2x the pilot's 5e-6 because the own-samples are ~2.5x shorter than the R1 traces (so a step carries far fewer supervised tokens and G2 needs a detectable MATH-500 gain), 2x under the ceiling because 2e-5 for 2 epochs on a model's own outputs invites the off-support format over-fitting that would defeat the F1 hypothesis; and at lr 1e-5 one AdamW update (~1e-5) is 12x SMALLER than a bf16 ULP at |w| ~ 2e-2 (~1.2e-4), so bf16 optimizer state would round most updates away (the pilot measured ~15x less learning that way).
• —NOTE — Qwen2.5 needs no completion-style workaround. The pilot's train_sft.py built targets completion-style because the DeepSeek template rewrites assistant content as content.split('</think>')[-1]. train_raft.py records tokenization.naive_render_check, which measures that Qwen2.5's template preserves the assistant content verbatim (the naive apply_chat_template([user, assistant]) render and ours agree, differing only by the template's trailing newline: e.g. 383 vs 382 tokens). The masking is verified rather than asserted: the manifest decodes the masked prefix (must end <|im_start|>assistant\n) and the supervised span (must be the completion + <|im_end|>), and the job aborts if labels != [-100] * n_prompt + input_ids[n_prompt:].
• —NOTE — tied embeddings. Qwen2.5-1.5B-Instruct sets tie_word_embeddings: true and its own model.safetensors has no lm_head.weight; the checkpoint writer drops it so the RAFT checkpoints are structurally identical to pipre's. `config.json`, `generationconfig.json, tokenizer.json and tokenizerconfig.json` are copied BYTE-IDENTICAL from the pinned pipre snapshot (pipre's config.json already says `torchdtype: bfloat16, usecache: true`), which is both correct and the strongest guarantee that pipost and pi_pre share an architecture.
• —OPERATIONAL — the local python had to be rebuilt off `/data`. After the container restart, /home/node/local/envs/eval-test was gone and uv's freshest CPython under /data/home/.local/share/uv/python was incomplete (an EMPTY lib/python3.12/re/ directory) while the intact 3.12.13 install had lost its executable bit — /data is a fuse mount that drops exec bits and is eventually consistent (edits there are also visible only a second or two later). Fixed by copying CPython 3.12.13 to /home/node/local/py312 (overlay fs) and rebuilding eval-test (harness pins) and a new raft-cputest (+ CPU torch 2.8.0) on it. Recorded because the next agent will hit the same thing.
• —VERIFIED — GATE G1 PASS. 6,400 prompts x k=4 = 25,600 samples from pipre; sample-level accuracy 0.3547 (9,081 correct); **4,187 kept** (pass@4 0.654) >> the 2,000 required. Acceptance by seed 2352 / 958 / 557 / 320 (the decay the "first correct in seed order" rule predicts; the four seeds' own accuracies are flat at 0.367 / 0.365 / 0.339 / 0.347, so the decay is selection, not a seed effect). `ncorrectofk histogram 2213 / 1518 / 1078 / 957 / 634. Extractors among kept: 3,478 answer_line + 709 boxed; verify methods 4,111 math_verify + 76 math_verify_cleaned. Rejections: 14,173 wrong_answer, 2,160 no_answer_extracted, 185 truncated_no_answer, 1 truncated_wrong_answer. Truncation over all samples 0.73 %. Kept completions mean 421 / p90 726 / max 1,690 tokens. Job 6a955ac80718b0f6d8908ae3, 30.2 min, 8,000 tok/s, $1.32. **Caveat for the report**: the keep rate falls monotonically with difficulty (0.894 at bin 3.0 -> 0.55-0.62 at bins 7.5-9.5), so the RAFT training set is easier-skewed relative to sft_train` — which is exactly what rejection sampling does, and worth stating rather than hiding.
• —VERIFIED — Stage 2 complete, every gate PASS (exit 0). 4,187 kept -> 3,977 train / 210 val; 124 steps (2 x 62), 3.35 M supervised tokens, 16.3 min of training, peak 30.2 GiB, max grad-norm 1.88. Train loss 0.1606 -> 0.0868. Validation 0.16881 -> 0.15517 (25 %) -> 0.15384 (50 %) -> 0.16230 (75 %) -> 0.16166 (100 %): the gate (final < step 0) passes, but the minimum is at 50 % = the end of epoch 1, i.e. the second epoch mildly over-fits. 2 epochs was FIXED BEFORE LAUNCH per section F2, so the root (100 %) is the pre-registered pi_post and no post-hoc checkpoint selection was done; checkpoint-50pct exists in the repo if the supervisor ever wants the lower-val-loss variant, and choosing it would be a new, logged decision. Shift sanity mean |delta logprob|/token 0.0986 (mean +0.0080, p50 0.0013, p90 0.293, max 5.00) over 15,994 tokens. Greedy generations coherent and on-format. Weights moved: global L2 2.60, 42.0 % of elements changed, max |delta| 9.77e-4 — a small, on-support shift. Model cmpatino/Qwen2.5-1.5B-Instruct-DeepMath-RAFT @ d46294c2a827c8558547c8ebac96a49b7a8410bb. Job 6a95624b0718b0f6d8908c05, 20.0 min, $0.87.
• —VERIFIED — TEACHER_POST contract satisfied. The repo root carries all five files run_opd.sh:284 fetches (config.json, generation_config.json, model.safetensors, tokenizer.json, tokenizer_config.json) plus vocab.json / merges.txt / README.md; model.safetensors is a single 3,087,467,144-byte file, byte-for-byte the same SIZE as pipre's (same tensor set, `lmhead.weight omitted under the tie); and all four non-weight files are sha256-identical to pi_pre's. Checkpoints live in checkpoint-{25,50,75,100}pct/, logs in logs/`. The driver's root-only fetch is unaffected by them.
• —VERIFIED — GATE G2 PASS. Paired raft_teacher - pre_teacher_ot, full protocol, unmodified harness, eval_model.paired_bootstrap_diff (10k resamples, defaultrng(42), percentile CI, two-sided bootstrap p), pairing on `sourceid: **MATH-500 sample4 (PRIMARY) 38.75 -> 49.95, gain +11.20 pp CI95 [+8.75, +13.55], p < 1e-4 — CI excludes 0, G2 PASS**; MATH-500 greedy 45.40 -> 53.80, +8.40 [+4.20, +12.60]; AIME24 sample32 2.19 -> 2.92, +0.73 [-1.04, +2.60] p=0.358; AIME25 sample32 0.42 -> 0.73, +0.31 [-0.62, +1.36] p=0.489 — both near-null, as section F3 pre-registered for this scale. **In-distribution sanity (ADDITION, no harness change — the harness's own teachereval` benchmark is the pilot pool's held-out 512-prompt split, strictly disjoint from `sfttrain, so it is a better in-distribution probe than a slice of the training prompts would have been):** sample4 34.52 -> 52.54, **+18.02 pp [+15.14, +20.90]**; greedy 37.11 -> 53.32, +16.21 [+11.52, +20.90]. Answer-format extraction success rose 0.918 -> 0.994 on teacher_eval and none`-extractor samples fell 187 -> 55 on MATH-500, so part of the gain is format compliance — but conditional-on-complete accuracy also rose (MATH-500 complete-only 39.10 -> 50.94), so it is not only format. Diagnostics: MATH-500 truncation 0.90 % -> 1.95 %, mean output tokens 716 -> 1,101; AIME24 6.5 % -> 11.6 % and 2,991 -> 4,634 (the RAFT model reasons longer on problems far outside its training difficulty), AIME25 4.8 % -> 8.7 %. Jobs 6a9567c40718b0f6d8908c89 ($3.29), 6a9567fd0718b0f6d8908c8f ($0.60), 6a9568100718b0f6d8908c91.
• —ADDITION (supervisor addendum 2026-08-31) — terminal `<|im_end|>` log-prob. Measured with train_raft.py --eos-probe, which reuses this file's own split/example construction so the 32 sequences are byte-identically the ones the training run held out. Mean log p of the terminal <|im_end|> at the TRUE end of 32 held-out completions: pi_pre -0.0804 (bf16) / -0.0741 (fp32); RAFT -0.0167 / -0.0154; delta = +0.0636 / +0.0587. The pre-registered prediction (RAFT stays in [-1, 0]) HOLDS, and the delta is not merely small but POSITIVE — the RAFT shift reward at the stop token slightly REWARDS terminating, where the corpus-SFT comparator OpenThinker3 measured -90.3 in Part A. This is the sharpest single discriminator yet between the RAFT pair and the corpus-SFT pairs, and it is the direct mechanism prediction of section F1. Job 6a956a240718b0f6d8908ccf, $0.06, eos_logprob_probe.json in the data repo.
• —ANOMALY (operational, cost) — the probe container did not exit. exp3-raft-probe's script printed DONE at 171 s and its running_secs froze at 347 s, but the platform kept the job RUNNING; the babysitter cancelled it on the 15-min silence rule at 19.7 min wall. CANCELED jobs report no durations, so the ledger carries two rows for that job id (0.094 h from the live running_secs, then a 0.234 h wall-clock correction) totalling the full 0.328 h / $0.82. The three later jobs exited promptly, so this looks like a one-off platform hiccup rather than a pattern — but it is the reason the silence window matters.
• —CORRECTION to the ANOMALY above — it was OUR bug, not a platform hiccup, and it is fixed. exp3-raft-sample-full did the same thing (script DONE at 30.2 min, running_secs frozen at 1,903 s, babysitter cancelled on the silence rule at 49.4 min wall). Both offenders are sample_raft.py; the train / eval / probe jobs all exited promptly. Root cause: sample_raft.py never shut the vLLM engine down, so its EngineCore worker held the container open — eval_model.py has always called release_engine() for exactly this reason. FIXED: sample_raft.py now calls release_engine() + collect_gpu() (the harness's verbatim shutdown-path policy) after the last chunk, before the manifest is written. Cost of the bug, booked conservatively on a wall-clock basis in two correction rows: $0.59 (probe) + $0.74 (full run) = $1.33. Anyone reusing sample_raft.py gets the fix; anyone writing a new vLLM script in this workspace should copy release_engine too.
• —RESULT — exp3 Part B COMPLETE ($8.23; ledger $295.32): G1 PASS — 4,187 own-samples kept of 25,600 (sample accuracy 0.355, pass@4 0.654; easier-skewed: keep rate 0.89 at difficulty 3 → 0.55–0.62 at 7.5–9.5; ground truth re-derived from DeepMath @ 5cf055d1 with 6400/6400 question + qhash match AND 6400/6400 pilot-target verification). SFT (lr 1e-5 cosine, 2 epochs = 124 steps, batch 64, fp32-master/bf16): val loss 0.1688 → 0.1617 with minimum at epoch 1 (mild 2nd-epoch over-fit; pre-registered 2 epochs kept, checkpoint-50pct preserved); shift 0.099 |Δlogp|/token. Model @ d46294c2a827c8558547c8ebac96a49b7a8410bb; tokenizer/config sha-identical to πpre; driver's 5-file fetch verified. G2 PASS — paired gains vs preteacherot: **MATH-500 sample4 +11.20 pp [+8.75, +13.55]** (greedy +8.40), teachereval in-dist +18.02, AIME24 +0.73 n.s., AIME25 +0.31 n.s.; extraction 0.918 → 0.994 but conditional-on-complete MATH-500 also +11.8 pp → not just format. EOS probe: π_pre −0.074, RAFT −0.015 (Δ +0.059) — training on its OWN complete samples slightly REWARDS stopping (vs OpenThinker3's −90.3). Anomaly: sampling jobs idled after DONE (missing vLLM release; $1.33 booked, script fixed). Local-python breakage after restart (fuse mount drops exec bits) fixed on local disk.
• —RESULT — anatomy RAFT column added ($0.38; identical frozen index verified, 2,960 sequences / 2,067,933 positions): RAFT pair (vs πpre, from stats.json) — token mean **−0.03** (RL −0.20, OT −0.23), student-weighted **−0.009**, **% tokens > 0 = 84.7 %** (RL 58.1, OT 34.3), **terminal EOS −0.006** (RL −6.3, OT −90.2), length-controlled AUROC 0.755 [0.716, 0.790]. The pre-registered P5 reading holds on the numbers ("qualitatively patterns with RL: near-zero terminal EOS + majority-positive tokens — in fact milder than RL on both"), though the coded P5 check is trivial (presence-only) — noted; the report cites the numbers. Two cosmetic issues flagged by the agent (a leftover placeholder column in summarytable.md; RAFT missing from the LINEAGE dict so its EOS caveat is mislabelled though the numbers are well-posed) — to be fixed in the report pass, stats.json unaffected.
• —GATE — H6a (exp3 smoke, job 6a9584b7, $1.08): step-1 deltaopd/weightedrewardmean = **−0.0048** (step 2 −0.0026) — strict "≥ 0" clause FAILS; the pre-registered weaker clause ("far closer to 0 than the OT pair's −0.0354, same πpre/student/config") HOLDS (7.4×). Adaptive KL reacting normally; clip 0.0; s/step 62.7–90.2. Env-only override path (OTTEACHERPOSTREPO/REV) verified by DRYRUN — driver unchanged @ b4dfdf37. Pre-registered GO → full run direct-opd-exp3-raft-full launched (7 h timeout, $60 ceiling, babysat). The decisive observables: H6b (clipratio < 0.5 through step 100 — the anti-EOS driver is gone for this pair) and H6c (paired student gain, MATH-500 primary).
• —CHECKPOINT — exp3 raft full run step 20 (job 6a958690): clip_ratio 0.000 at all 20 steps (B at the same step: 1.00 since 15), resp 339 → 431 (no trend to the cap), weighted reward −0.0048 → −0.0020 (stable near zero, no ratchet), KL 2.5 → 2.045, s/step 69.0 → projected ≈ 2.0 h / ≈ $20. H6b trending PASS.
• —RESULT — H6b PASSES (exp3 raft full run, job 6a958690): 100/100 steps, training exit 0 after 8,288 s; clipratio mean **0.0004**, collapse-watch "first step > 0.5: none"; responselength 339 → 500 (mean 523; cap 3,328 never approached); weighted reward stable −0.005 → −0.003; KL 2.475 → 0.915 adaptive. The first SFT-pair Direct-OPD run in the campaign that did not collapse — same student, same config, same πpre as condition B (collapsed at step 12); the only change is πpost's provenance (own-samples RAFT vs corpus SFT). Merge/upload in progress; H6c (paired student gains, MATH-500 primary) next: ckpt-100 terminates → FULL protocol applies.
• —VERIFIED — exp3 raft run close-out (job 6a958690, COMPLETED, 9,232 s, $25.64; ledger $322.42): checkpoints 20–100 + root + logs at cmpatino/Qwen2.5-7B-Instruct-DirectOPD-RAFTShift-100 @ de65f5a389be54f1443810a248c9e2d58c39f3bd, driver verify 0 problems; max clip over 100 steps 0.0078; logratiopos_frac note: verl's metric reads ~0.08 on TRAINING rollouts (top-16 candidate basis) vs the anatomy's 84.7 % realized-token basis — different quantities, flag for the report. H6a partial, H6b confirmed; H6c to the eval wave.
• —DECISION (supervisor) — H6c eval wave: ckpt-100 × {math500, aime24, aime25} at the FULL protocol (terminating model, ≈ $4–6) + MATH-500 sample4 curve over ckpt-20/40/60/80 (≈ $6) + AIME24 8-sample curve over the same (≈ $2) — all vs student_init, paired. ≈ $12–15 total.
• —VERIFIED — 11-job H6c eval wave, all COMPLETE, exit 0, all pushed. Target cmpatino/Qwen2.5-7B-Instruct-DirectOPD-RAFTShift-100 @ de65f5a389be54f1443810a248c9e2d58c39f3bd. Jobs: ckpt-100 x {math500, aime24, aime25} FULL protocol (opd_student_raftshift); ckpt-20/40/60/80 x math500 FULL protocol + x aime24 8-sample curve (opd_student_raftshift-ckpt{20,40,60,80}). Total cost $9.94 (jobs 6a95ac1d/6a95ac3a/6a95ac6c/6a95ac86/6a95ac97/6a95acb0/6a95acc4/6a95ace8/ 6a95acfe/6a95ad23/6a95ad32), ledger now $332.36 / $500 ($167.64 remaining). Model TERMINATES cleanly as H6b predicted: truncation 0.05–2.5% everywhere (max output tokens well under cap on all passes), no unit hit the 31,744 cap in any meaningful fraction of samples — costs landed at the student_init-like $0.5–1.4/job estimate, not the R1shift/OTshift non-terminating $5–12/job regime.
• —Acceptance checklist PASS on all 11 units: resolved_sha = pinned de65f5a3... everywhere; deviations=[] on all 7 full-protocol units, and exactly the expected "8 samples seeds 0..7 ... DIAGNOSTIC curve protocol" deviation on the 4 aime24 curve units; chat_template_sha256 = cd8e9439... on every unit; rendered_tail_example ends <|im_start|>assistant\n; sample counts correct (math500 greedy n=500/sample4 n=2000, aime24/25 full n=960, aime24 curve n=240); benchmark.parquet_sha256 verified byte-identical to student_init's on the matching benchmark for every one of the 11 units (independent check, since the panel-wide aggregate_evals.py --regrade alignment audit reports ok=False — that failure is entirely attributable to OTHER conditions' _sub100/8-sample diagnostic units sharing the results repo, exactly the "nested grids, expected" case the auditor itself names; every opd_student_raftshift* <-> student_init id-set match was exact with zero warnings in the paired-gains script).
• —RESULT — H6c FAILS. Paired bootstrap (10k resamples, default_rng(42), percentile CI, two-sided p, eval_model.paired_bootstrap_diff, pairing on source_id/unique_id), full table in supervisor/h6c_gains.json: - ckpt-100 MATH-500 sample4 (PRIMARY): −1.65pp [−3.45, +0.15], p=0.065 — CI does NOT exclude 0, point estimate is slightly NEGATIVE. H6c verdict: FAIL. - ckpt-100 MATH-500 greedy: −0.80pp [−4.00, +2.20], p=0.575. - ckpt-100 AIME24 sample32: −1.56pp [−4.17, +0.83], p=0.179. - ckpt-100 AIME25 sample32: −2.19pp [−5.21, +0.00], p=0.035 — CI upper bound touches exactly 0, essentially a significant regression, not a gain. - Curve MATH-500 sample4 (full protocol) vs studentinit: ckpt20 −0.70pp, ckpt40 +0.25pp, ckpt60 −0.35pp, ckpt80 −0.95pp — every CI straddles 0, no monotonic trend with training step. - Curve MATH-500 greedy: ckpt20 +0.60, ckpt40 +0.60, ckpt60 −1.80, ckpt80 −1.60pp — all n.s. - Curve AIME24 8-sample (vs studentinit seeds 0-7): ckpt20 −2.50, ckpt40 −2.08, ckpt60 −1.67, ckpt80 −2.08pp — all n.s., all negative point estimates. - Every one of the 11 paired comparisons (ckpt-100 x 4 benchmarks/passes + 4 checkpoints x 3 pass/benchmark combos) has a non-positive or non-significant point estimate; none has a CI excluding 0 on the positive side. No checkpoint, benchmark, or pass shows a detectable gain. - Diagnostics: truncation 0.05–2.5% throughout (no cap-limited units); mean output tokens stable 546–1987 depending on benchmark/checkpoint, growing mildly with training step on some passes (math500 greedy tokmean 715->838 ckpt20->ckpt80; aime24 tokmean ~1500-2000, no clear trend); extractor split answerline-dominant (e.g. ckpt-100 math500 sample4: 1624 answerline / 351 boxed / 25 none) with a low and stable none fraction (~1-2% math500, ~7-8% aime — no format degradation with training step).
• —INTERPRETATION — pre-registered failure read applies. F3: "H6a,b true but H6c false -> the channel is safe but this shift is too small to detect." H6a was the weaker-clause PASS (RAFT pair's step-1 reward −0.0048, 7.4x closer to 0 than OT's −0.0354, but still negative, not literally >= 0). H6b PASSED cleanly (max clip_ratio 0.0078 over 100 steps, no collapse). H6c now FAILS: the RAFT shift never collapsed the student (unlike every corpus-SFT condition in exp2/2b), but it also never measurably moved held-out capability — consistent with H6a's own sign (a small residual on-support-but-still-negative reward has nothing positive to transmit). This is the cleanest evidence yet for the F1 mechanism claim in its safety half (no collapse when the shift is on-support) while leaving the transfer half unconfirmed at this shift magnitude — the RAFT teacher's own gain was real (G2 PASS, MATH-500 +11.2pp) but the shift the student received from it was apparently too weak/diffuse (own-sample RAFT SFT with |dlogp|/token mean 0.099, far smaller than the corpus pairs') to detect in a 7B student over 100 Direct-OPD steps.
• —SYNTHESIS (supervisor, post-H6c) — exp3 outcome: H6a partial (step-1 reward −0.0048, 7.4× closer to 0 than the corpus pair; still negative), H6b TRUE (first non-collapsing SFT pair: max clip 0.0078 over 100 steps), H6c FALSE (ckpt-100 MATH-500 −1.65 [−3.45, +0.15]; all 16 gain cells flat-to-negative; no trend with step). The complete two-axis picture: on-supportness governs stability, shift magnitude governs transfer. Corpus SFT = large-magnitude, off-support (|Δ| ≈ 2.7/token, anti-EOS) → collapse, style-only. RAFT (124 steps on 4,187 own samples) = on-support but tiny (|Δ| ≈ 0.099/token) → safe, null; the pilot's gain-per-shift (+8.75 pp per unit RMS) predicts < 1 pp here — below CI resolution, consistent with the measurement. RL = on-support AND large (pilot RMS 1.02) → transfers (~half the teacher gain), because for KL-regularized RL the log-ratio is the learned advantage, so magnitude and usefulness scale together. Open (stated as such): whether iterated/scaled RAFT — which is one step of an RL-like loop — interpolates to the RL case.

3. Setup

3.1 Pinned revisions

3.2 Roles

The Direct-OPD reward is the token-level log-ratio log πpost(t) − log πpre(t), scored on the student's own rendered token ids (input_tokenizer = null, as the method prescribes). πpre and πpost are therefore not policies to imitate — they are the two ends of a difference. A property worth stating early, because it frames everything in §5: on MATH-500 the initial 7B student already matches π_post; on AIME it is far ahead of π_pre but well behind π_post. The teacher pair supplies a shift signal, not a uniformly better policy — the exact numbers are in §5.

3.3 Training configuration

★ changed from the pilot's config: student, teacher_ref, MAX_PROMPT_LENGTH, MAX_RESP_LENGTH, PPO_MAX_TOKEN_LEN_PER_GPU. Everything else is the pilot's pinned configuration. Memory knobs (optimizer_offload, the log-prob token budgets, GPU_MEMORY_UTILIZATION) are operational, not scientific, and were adjusted after the smoke without a deviation entry — the pre-registration says so explicitly.

3.4 Evaluation protocol

Curve and reduced-protocol units use 8 samples/problem (seeds 0–7), which is the pre-registered diagnostic-curve protocol. Wherever an 8-sample unit is compared with student_init, this report restricts student_init to the same eight seeds of its 32-sample unit, read from its generations.parquet — so the comparison is paired in problems and matched in samples. The naive comparison against the full 32-sample mean is reported beside it, labelled.

4. Phase 0 gates

All Phase 0 gates ran on CPU, at $0, before any GPU spend. Both artifacts are in MANIFEST.json.

A known, reported-not-fixed method property: special IDs 151643–151649 mean different things in the DeepSeek πpost than in the Qwen πpre/student. Only the response's terminal <|im_end|> is affected — about one token per response.

5. Teacher gains — the premise

This is the section the whole experiment is built on: the shift we are asking Direct-OPD to carry must actually encode held-out capability. Paired by problem, percentile bootstrap over problems, 10,000 resamples, seed 42.

The pre-registered TEACHER-GAIN GATE passes: +24.375 [+15.417, +34.167] p<0.0001 on AIME 2024. The premise is true — this teacher pair is separated by a large, real, held-out capability difference, which is exactly what the pilot's SFT pair lacked.

Two caveats that must travel with these numbers.

1. 1.π_pre is measured at a 3,200-token cap it cannot exceed, and truncates heavily there — see the rates in §8.1. Part of the measured teacher gain is a token budget difference, not a reasoning difference. §8.4 bounds it from π_post's own stored generations, and the bound is not small:

• —AIME 2024 — native +24.375 [+15.417, +34.167] p<0.0001 → at π_pre's budget, over-cap-as-wrong +3.646 [-1.042, +9.167] p=0.1426.
• —AIME 2025 — native +21.146 [+10.208, +33.333] p<0.0001 → at π_pre's budget, over-cap-as-wrong +6.354 [+0.625, +13.333] p=0.0216.
• —MATH-500 `sample4` — native +46.500 [+43.200, +49.850] p<0.0001 → at π_pre's budget, over-cap-as-wrong +31.550 [+27.950, +35.050] p<0.0001.

On AIME the restricted gain's CI includes zero; on MATH-500 it does not. The pre-registered gate is stated on the native protocol and it passed — but any statement of the form "the shift carries N pp of capability" should quote the range, not the headline.

1. 1.The teacher pair is not better than the student. On MATH-500 sample4 the initial student scores 74.35 [71.15, 77.55] against πpost's 75.20 [72.20, 78.15] — statistically indistinguishable. On AIME 2024 the initial student scores 12.19 [3.96, 22.50] against πpost's 29.06 [18.33, 40.73]; πpost is ahead here, but the student is far ahead of πpre (4.69 [1.46, 9.06]). So the shift is a direction, not a better policy to copy — the same framing caveat the pilot recorded, and it matters for reading §7.

6. Training dynamics

100/100 steps on a100x4, 100 metric records, 148 numeric keys per step. The run splits cleanly into three regimes. The collapse band is steps 58–85; the first step whose length-clip ratio exceeds 0.5 is step 61.

Per-step snapshots at the checkpoint steps and either side of the break:

[image] [image] [image] [image] [image]

The mechanism, read off the metrics

1. 1.The shift reward was negative on the student's native style, for the entire pre-collapse regime (mean -0.0043, never above -0.0022 in steps 1–57). Read literally: on the tokens this Qwen2.5-Instruct student actually likes to emit, πpost assigns *less* probability than πpre. The objective's only instruction was therefore "stop writing like yourself" — it never pointed at a better answer, only away from the current one.
2. 2.The KL anchor gave way under that pressure. The adaptive controller ratcheted the coefficient from 2.5 down to 1.064 across the collapse band — a negative-reward regime pushes the controller to loosen the leash exactly when the leash is the only thing holding the policy in a sane region.
3. 3.The policy escaped to where the two teachers agree. As length exploded, the weighted reward went to ≈0 — not positive, zero. A log-ratio of zero means πpost and πpre assign the same probability, and the cheapest such region for a language model is degenerate repetition. Sampled rollouts show it exactly: an R1-style opening ("Okay, I need to solve…") followed by an endless Answer: <x> loop.
4. 4.Everything else follows. Entropy 0.352 → 0.058; pg_loss → ~0; grad norm → ~0.25; s/step 4×'d because every rollout ran to the cap. The late partial recovery (steps 86–100: entropy back to 0.667, weighted reward turning +0.0026) is real in training, at the 3,328-token training cap. It does not carry over to eval-time sampling at the 31,744-token cap: the P5 probes (logs/p5_repricing.md) measured 87.5–100 % truncation at T=0.7 and greedy for ckpt-60, ckpt-80 and ckpt-100 alike — which is what forced the reduced protocol.

7. Student results — the primary endpoints

7.1 Comparability caveat (read this before the table)

The ckpt-100 endpoint is measured under the cost-forced reduced protocol of §2.1(3). Concretely:

• —AIME 2024 / 2025 — ckpt-100 at 8 samples/problem (seeds 0–7). The primary comparison restricts student_init to the same 8 seeds of its 32-sample unit, recomputing its per-problem means from generations.parquet. Both sides then have identical problems and identical seeds; the CI is wider than a 32-sample comparison would give, and that is the honest price of the reduction.
• —MATH-500 — ckpt-100 on a 100-problem evenly-spread subset, with the full pass set (greedy + sample4). The comparison restricts student_init to the same 100 unique_ids. The sample grid is not reduced here — only the problem set.
• —The naive comparison (8 ckpt-100 samples vs student_init's full 32-sample means) is given beside the primary one and labelled unpaired-in-samples. It is not the pre-registered estimand; it is there so a reader can see the reduction did not manufacture the result.

7.2 Primary endpoints

7.3 Transfer ratio (pre-registered guard)

The transfer ratio is reported only when all four guards pass: \|teachergain\| ≥ 1.0 pp, teachergain > 0, its 95 % CI excludes 0, and student_gain > 0. This is not decoration — the pilot's measured teacher gain on AIME 2024 was negative, and without the sign guards two regressions divide into a tidy "ratio 1.0".

A note on the guard, offered as a methods observation rather than a result. The pre-registration (inherited from the pilot) guards the sign of both gains and the significance of the denominator. It does not guard the significance of the numerator — and the AIME 2025 row is exactly the case that exposes the gap: a nominally positive but statistically null student gain divided by a large real teacher gain yields a small, tidy, entirely meaningless "transfer ratio". The value is printed because the rule says to print it, and the numerator CI excludes 0? column is printed beside it so nobody can read it as partial transfer. A future pre-registration should add that fourth significance test.

7.4 The checkpoint curve

Every point is a paired per-problem bootstrap against a comparable student_init slice (same seeds on AIME; same problems on MATH-500). The collapse band (steps 58–85) is shaded in the figure.

[image]

AIME 2024, 8 samples/problem (seeds 0–7); baseline = student_init restricted to the same seeds

MATH-500 `sample4` (primary pass) — MATH-500 sample4; steps 20/40 on all 500 problems, step 100 on the 100-problem evenly-spread subset (NOT the same problem set)

MATH-500 `greedy` (secondary pass) — MATH-500 greedy; steps 20/40 on all 500 problems, step 100 on the 100-problem evenly-spread subset (NOT the same problem set)

AIME 2025 (replication); only the ckpt-100 endpoint was measured

7.5 Headline table

One row per benchmark. Teacher gain on the native protocol; student gain at ckpt-100 under the reduced protocol; both are paired percentile bootstraps over problems (10,000 resamples, seed 42).

8. Diagnostics

8.1 Truncation, length and answer format, per unit

answer_line / boxed / none are the three branches of the pre-registered extractor (last Answer: line, else last \boxed{}, else nothing). A none sample is graded incorrect by construction.

8.2 Answer-format split, side by side

The same numbers as §8.1, arranged so the direction of the style change is visible. Primary pass only.

Read the AIME rows first, where the effect is largest. Both teachers lean `\boxed{}` there (61–73 % of samples) — πpre because its own chat template injects *"put your final answer within \boxed{}"*, πpost because the R1 distillation writes that way — and the initial student splits roughly 58/38 between the two formats. The OPD student goes almost purely to the prompt-instructed `Answer:` line and extinguishes `\boxed{}` altogether (37.5 % → 0.8 % by step 20, 0.0 % by step 40). On MATH-500 π_post itself prefers Answer: (79.8 %) and the initial student already does too (83.7 %), so there is less room to move — and the student still moves past both, to 98 %.

The direction is the point. On the benchmark where the two teachers agree on \boxed{}, the student runs the other way. Whatever the token-level log-ratio rewards, it is not "emit π_post's surface form" — the pilot saw the same non-imitation in both of its arms, and it reproduces here with a completely different teacher pair and a 4.7× larger student.

8.3 Extraction on truncated samples — new for this experiment

The pilot could treat truncation as "no answer". This experiment cannot. Two independent reasons, and they point in opposite directions:

• —π_pre is cut off at 3,200 tokens after it has often already emitted a \boxed{} or an Answer: line, so a large share of its truncated samples still extract — the answer is there, the work around it is not.
• —The collapsed OPD checkpoints do the opposite in kind and the same in effect — and this is now measured, not predicted. The degenerate policy reaches an answer and then loops on it, so a truncated sample still ends in a gradeable Answer: line. At ckpt-100: 93.2 % of the 221 truncated AIME 2024 samples still extract an answer (and 9.50 pp of them are correct), and on the MATH-500 subset 98.2 % of the 386 truncated sample4 samples extract, scoring 69.95 pp. The collapse is not an answer-extraction failure. The model still solves the problem; it just never stops talking about it. That is the single most useful diagnostic in this report: it separates "the policy was destroyed" from "the policy lost its stop token", and the evidence is for the latter.

Either way, truncation and non-extraction have come apart and both must be reported. Measured now:

Two readings of the table. π_pre: 40–47 % of its truncated samples still extract, against 0–12 % for the two teachers' and the initial student's — so reading "truncated ⇒ no answer" would understate π_pre and, through it, overstate the teacher gain. The collapsed checkpoints: 84–98 % extraction on truncated samples, with real accuracy behind it. Truncation rate alone is therefore a terrible summary of what went wrong here, and this column is what replaces it.

8.4 Sensitivity view — πpost at πpre's cap

πpre is capped at 3,200 tokens by its own trained context, while πpost runs to 31,744. Part of the §5 teacher gain is therefore a budget difference rather than a capability difference. This view costs no new compute: it restricts π_post's stored generations to samples of ≤ 3,200 output tokens and re-scores.

Two readings, because neither alone is honest:

• —kept-only — accuracy on the π_post samples that fit under 3,200 tokens. This is conditional accuracy and is biased upwards: short samples are the easy problems.
• —over-cap-as-wrong — every π_post sample longer than 3,200 tokens counted incorrect. This simulates the cap and is a lower bound: a truncated sample sometimes still extracts a correct answer (§8.3).

The true "π_post at 3,200 tokens" lies between them. Both are labelled sensitivity views and neither replaces the §5 primary numbers.

The last column is why the kept-only view must not be quoted on AIME: on the 30-problem sets only ~9–11 problems have any π_post sample under 3,200 tokens, and they are the short — i.e. easy — ones. The kept-only row is included for completeness and is not a usable estimate there. On MATH-500, where ~70 % of samples survive the cut and 347–411 of 500 problems are represented, it is informative.

Restated as gains against π_pre's native numbers (paired on the problems each view covers):

The row that matters is `aime24` / over-cap-as-wrong. It is the pre-registered teacher-gain benchmark, measured with both models on the same token budget and the unmeasurable cases resolved against π_post, and it is the one place where the premise becomes uncertain. MATH-500 keeps a large, budget-independent gain, which is why the co-primary design was worth having.

9. Interpretation vs the pilot

The pilot's finding was that Direct-OPD is a faithful courier of whatever the shift encodes: an SFT shift that encoded narrowing produced a student that narrowed. The obvious follow-up — give it a shift that encodes real capability — is this experiment. The answer is negative, and the interesting part is why. With a teacher pair that demonstrably carried held-out capability, 100 Direct-OPD steps produced a student that is no better on any benchmark at any checkpoint, significantly worse on MATH-500, and — after step ~61 — unable to terminate at all. The pilot's verdict (style transfers, capability does not) survives the strongest test we could give it, and a new failure mode appears alongside it.

What we can say, in decreasing order of confidence.

1. 1.The premise is no longer the limiting factor, and the result did not follow. The teacher gain is large and unambiguous (+24.375 [+15.417, +34.167] p<0.0001 on AIME 2024). The student's gain, at every checkpoint measured, is not. Whatever blocked transfer here, it is not "the shift had nothing to give" — which was the pilot's explanation and is now excluded.
2. 2.Surface form moved; capability did not — and the surface form did not move towards π_post. By step 20 the student had abandoned \boxed{} almost entirely in favour of the prompt-instructed Answer: line, while on AIME both teachers lean \boxed{} (§8.2). Accuracy over the same interval is flat on AIME and slightly negative on MATH-500 (§7.4). This is the pilot's finding 5 reproduced with a different teacher pair, a different student family and a 4.7× larger student: a token-level log-ratio objective buys a change of surface form first, and the form it buys is not the post-teacher's. The natural reading is that the log-ratio is largest on tokens where the two teachers disagree about style, and the cheapest way to increase it is to leave the region where they disagree — not to imitate either one.
3. 3.The failure mode is termination collapse — a lost stop token, not a destroyed policy. It is not a generic "OPD diverged". The chain is legible in the metrics (§6): a reward that is negative on the student's native style → an adaptive KL controller that loosens under sustained negative reward → escape into the degenerate region where πpost ≈ πpre → an Answer: repetition loop that never emits a stop token. It is worse at eval time than in training — at the 31,744-token cap ckpt-100 truncates on 92–100 % of samples despite its training-time partial recovery at the 3,328-token cap — and §8.3 pins down what "truncated" means here: 84–98 % of those truncated samples still extract an answer, and on MATH-500 ~70 pp of them are correct. The model still reasons; it cannot stop. This distinction matters for the fix list below: three of the five candidates target termination specifically.
4. 4.A capability-bearing shift is not sufficient; distributional proximity may be necessary. The two teachers here are far apart in style (a \boxed{}-instructed math base vs an R1 <think> distillation) and both are far from the student (non-thinking Qwen2.5-Instruct, 4.7× larger). The log-ratio is then dominated by style disagreement, and the capability difference — the thing we wanted — is a small residual underneath it. This is a hypothesis this experiment suggests; it does not test it.

What would need to change, concretely, for a rerun to be informative.

A rerun that changes only one of these is worth more than one that changes all five: the first three test the collapse, the last two test the premise about distance.

10. Limitations

1. 1.The primary endpoint is measured under a reduced protocol (§2.1(3), §7.1). It is the same estimand with wider intervals, not a different one — but it is not the pre-registered sample count, and a null result at 8 samples/problem is weaker evidence than a null at 32.
2. 2.MATH-500's ckpt-100 unit is a 100-problem subset, so its interval is roughly √5 wider than the 500-problem baseline's, and it is not the same problem set as the ckpt-20/40 curve points. The curve table says so on every row.
3. 3.π_pre is context-limited at 3,200 tokens and truncates heavily, so the teacher gain is partly a token-budget difference. §8.4 bounds it but cannot eliminate it: on the pre-registered gate benchmark the bound runs from +3.646 [-1.042, +9.167] p=0.1426 (πpost held to πpre's budget, over-cap counted wrong) up to +24.375 [+15.417, +34.167] p<0.0001 (native protocol). The honest statement of the premise on AIME 2024 is that range, not the headline point. MATH-500 is where the premise survives the correction intact.
4. 4.n = 30 on AIME. Even at 32 samples/problem the per-problem bootstrap over 30 problems gives intervals ~10 pp wide. MATH-500 is the co-primary precisely because 500 problems give ~4× tighter intervals — which is why losing the full MATH-500 endpoint to cost hurts.
5. 5.One run, one seed (42), one condition. No RL comparator, no repeat at a different seed, no ablation of the collapse. We cannot separate "Direct-OPD cannot carry this shift" from "this configuration collapsed before it could".
6. 6.The collapse confounds everything after step ~58. ckpt-60/80/100 are measurements of a degenerate policy. Their accuracies are real, but they answer "what does a collapsed policy score", not "how much capability transferred".
7. 7.No shift-magnitude diagnostic. The pre-registration folded the reward-sanity check into the P3 smoke to save cost, so there is no weighted-RMS shift magnitude for this pair and therefore no gain-per-unit-shift number comparable to the pilot's.
8. 8.π_post's AIME units are imported from the pilot, not re-run. Same model revision, same harness, same protocol, verified by provenance.json per unit — but not an independent measurement.
9. 9.Cost basis. The quoted total is wall-clock occupancy × flavor rate including aborted work; the platform's running_secs basis is a floor (see §11).

11. Cost & reproduction

$332.36 of a $200 cap ($-132.36 unspent), recomputed by summing cost_usd over all 83 rows of ledger/cost_ledger.csv.

By hardware: a100x4 $176.73 · a100-large $155.63 · cpu-basic $0.00.

⚠️ Two cost bases. wall-clock occupancy x flavor rate from ledger/costledger.csv, INCLUDING any aborted work — the same (larger) basis the pilot quoted. The platform's own `runningsecs x rate` is a floor, because CANCELED jobs report running_secs = 0 regardless of real occupancy. This report quotes the larger basis, as the pilot did; nothing in the decision record turns on the difference.

⚠️ This ledger is cumulative over run 1, the exp2b extension AND exp3. The cap was raised $200 → $500 by the user on 2026-08-26. Run 1's own phases (P1–P6) total $92.87, inside the original $200 cap; the exp2b extension's *b phases (P2b, P3b, P4b, P5b) add $192.02; exp3's phase (P7) — the shift anatomy, the RAFT pair and its whole eval wave — add $47.47, for $332.36 of $500. §14.4 gives the breakdown, including the cancelled runs booked on the wall-clock basis.

Ledger reconciliation: every eval unit on the Hub has a matching cost-ledger row (units imported from the pilot repo excepted, which cost nothing).

Imported at $0 (no GPU spend expected, provenance.json in each unit): post_teacher/aime24, post_teacher/aime25.

11.1 Reproducing the whole campaign

Everything below is pinned by revision or by sha256, and every table in this subsection is generated from its source at build time (eval_model.MODEL_REGISTRY, ledger/resolved_revisions.json, ledger/staged_driver.json, the files on disk) rather than typed in. A third party with the results repo and this list can rebuild the campaign end to end.

(a) Models — the fixed inputs. Each alias is exactly the --model the harness was launched with.

(b) Models we trained. Every Direct-OPD run in the study, with the repo and revision its checkpoints live at. A cancelled run published nothing — that is why its evidence in §13.5 is training-only.

(c) Datasets and the method itself.

(d) Code, by content hash. The driver was staged into the results repo before every launch, so a run's exact driver is recoverable from the revision it was staged at.

(e) The launches themselves. The exact hf jobs run command line of every training launch is stored verbatim, together with the babysitter's per-step tables — which are the only surviving evidence for the three cancelled conditions, since they published no checkpoint. Every one of these files is listed in MANIFEST.json with its sha256 and is uploaded to the results repo under logs/ alongside this bundle:

logs/full_launch_command.txt
logs/launch_smoke_stdout.txt
logs/redirect_launch_commands.txt
logs/exp2b_full_lowlr_steps.tsv
logs/exp2b_full_openthinker_klfloor_steps.tsv
logs/exp2b_full_openthinker_qwen3_4b_steps.tsv
logs/exp2b_full_openthinker_steps.tsv
logs/exp2b_full_short_steps.tsv
logs/exp3_full_raft_steps.tsv
logs/full_steps.tsv

Evaluation launches follow one form (the runbook is code/eval/LAUNCH.md); the reduced-protocol variants differ only in --num-samples/--seeds (AIME) or --limit-problems (MATH-500) and are pushed under their own _sub8 / _m500sub100 aliases so they can never masquerade as canonical units:

# one eval unit
hf jobs uv run -d --image huggingface/trl --secrets HF_TOKEN --flavor a100-large \
  -e HF_HOME=/tmp/hf -e HF_HUB_ENABLE_HF_TRANSFER=1 \
  --label experiment=direct-opd-sft-transfer --label run_id=<run_id> \
  eval_model.py --model <repo-or-alias> --revision <sha> [--subfolder checkpoint-N] \
    --model-alias <alias> --benchmark <aime24|aime25|math500> \
    --output-dir /tmp/exp2-out --skip-if-exists --push

Each training run's own verl artefacts (hydra_config.yaml, hydra_overrides.yaml, run_manifest.json, metrics.jsonl, the gzipped train log) are in the model repo of that run under logs/, and the four 2-step smokes are in the results repo under phase4/smoke/<condition>-<timestamp>/.

(f) The ledger. ledger/cost_ledger.csv — append-only, one row per job, keyed by job id. Every figure in §11 and §14.4 is summed from it at build time, so booking a late job and re-running the build is all it takes to correct the cost of this report; nothing about the cost is frozen into the text.

11.2 Rebuilding this bundle

export HF_HOME=/home/node/local/hf-cache          # HF_TOKEN must be in the environment
/home/node/local/envs/eval-test/bin/python \
  /data/workspaces/direct-opd/exp2-sft-transfer/code/report/build_report.py

# reuse the local cache instead of re-downloading from the Hub:
/home/node/local/envs/eval-test/bin/python \
  /data/workspaces/direct-opd/exp2-sft-transfer/code/report/build_report.py --offline

# the canonical cross-model aggregate (this report reuses its numerics)
/home/node/local/envs/eval-test/bin/python code/eval/aggregate_evals.py --from-hub \
  --results-repo cmpatino/direct-opd-sft-transfer-results \
  --curve-ckpt100-from-primary --output-dir artifacts/eval-agg

The build is idempotent and CPU-only: it downloads (or reuses) every eval unit and every training log, recomputes every statistic from the raw per-problem and per-sample data, regenerates all nine figures, and rewrites aggregate.json, README.md, summary.md, MANIFEST.json and CHECKSUMS.sha256. No number in this report is hand-copied between artifacts, and no number is quoted that does not come from a file listed in MANIFEST.json. Re-running it after the 0 pending unit(s) land is the only action needed to finish the report — every placeholder becomes a measured value and nothing else changes.

12. Artifact inventory

12.1 Evaluation units

Sample-level alignment audit (aggregate_evals.audit_alignment, run per comparison group): OK.

• —group full: ok=True
• —group curve8: ok=True
• —group sub100: ok=True
• —group exp2b-full: ok=True — aime24: grids differ but are NESTED {'opdstudentotshift-short-ckpt10': 240, 'opdstudentotshift-short-ckpt4': 960, 'opdstudentotshift-short-ckpt8': 960, 'opdstudentotshiftlowlr': 240, 'opdstudentotshiftlowlr-ckpt20': 960, 'opdstudentotshiftlowlr-ckpt40': 960, 'opdstudentr1shift-ckpt40-full': 960, 'postteacherot': 960, 'preteacherot': 960} — expected when diagnostic curve units (8 samples/problem) sit beside primary units; aime25: grids differ but are NESTED {'opdstudentotshift-short-ckpt10': 240, 'opdstudentotshift-short-ckpt4': 960, 'opdstudentotshift-short-ckpt8': 960, 'opdstudentotshiftlowlr': 240, 'opdstudentotshiftlowlr-ckpt20': 960, 'opdstudentotshiftlowlr-ckpt40': 960, 'opdstudentr1shift-ckpt40-full': 960, 'postteacherot': 960, 'preteacherot': 960} — expected when diagnostic curve units (8 samples/problem) sit beside primary units
• —group unexpected: ok=True
• —group exp2b-curve8: ok=True
• —group exp3-full: ok=True
• —group exp3-curve8: ok=True

12.2 This bundle

aggregate.json        every statistic in this report, machine-readable
summary.md            the same, rendered as tables (also pushed to evals/aggregate/)
README.md             this file
figures/f1..f5*.png   Phase-4 training figures, reused verbatim from artifacts/figures/
figures/f6*.png       student gain vs OPD step, paired CIs, collapse band
figures/f7*.png       teacher gain vs student gain
MANIFEST.json         every input file, its sha256 and its repo revision
CHECKSUMS.sha256      sha256 of every file in this bundle

MANIFEST.json lists 186 input files. Large inputs that already live in the results repo (the generations.parquet of every unit) are referenced by repo + revision + sha256, not duplicated here.

13. Extension exp2b — a second teacher pair, three cancelled conditions, two redirects

Pre-registered PREREGISTRATION.md sections E1-E3 (2026-08-26), extended by §E5 (2026-08-28). Budget cap raised to $500 cumulative. Sections 0–12 above are run 1 and are unaffected by anything here.

Status: 32 of 32 live exp2b units are on the Hub, and a further 33 pre-registered units are canceled or dropped — they can never exist, because the runs that would have produced them were stopped before their first checkpoint save (§13.5) or the tier was dropped by the user. Every cell below is a measured value; nothing here is an estimate and nothing is outstanding.

13.1 The new teacher pair, and why it needs no cap deviation

• —π_pre = Qwen/Qwen2.5-1.5B-Instruct @ 989aa7980e4cf806f80c7fef2b1adb7bc71aa306
• —πpost = `open-thoughts/OpenThinker3-1.5B @ 0ee90a38b29bfac8b8b005da9ae32c59e2943785` — **SFT-only** (7 epochs on OpenThoughts3-1.2M) from that same πpre, i.e. the same family and post-training lineage as the 7B student.
• —shift key openthinker3_sft

Both models have maxpositionembeddings 32,768 and the longest prompt under their (identical) chat template is 872 tokens, so 872 + 31,744 = 32,616 fits and neither carries a cap deviation — unlike run 1's pi_pre.

This is the single biggest measurement difference between run 1 and exp2b: run 1's AIME teacher gain shrank from +24.38 pp to +3.65 [−1.04, +9.17] once πpost was held to πpre's 3,200-token budget (§8's cap-sensitivity view). The exp2b pair has no such confound — both models are measured at the same, full, pre-registered cap.

13.2 Teacher gains for the new pair — the exp2b premise

TEACHER-GAIN GATE (AIME 2024): PASS — +51.771 pp [+38.646, +64.375].

PREREGISTRATION E3: paired post-pre on AIME24 must be > 0 with a 95 % CI excluding 0 (expected ~ +49 pp from the published 3.0 -> 52.0)

These are the paired gains the gate is stated on. They are what makes the rest of this section a result rather than a null: the shift this pair encodes is real and large, so when four separate students fail to inherit any of it, the explanation cannot be that the shift had nothing to give.

13.3 A1 — run-1 ckpt-40 at the FULL protocol

ckpt-40 is the last pre-collapse checkpoint of run 1 and it terminates (0.4 % truncation). It was only ever measured on the 8-sample curve protocol; at the full protocol it becomes the study's one ckpt-vs-init comparison that needs no caveat at all — same 32 seeds, same 500 problems, on both sides.

Alias opd_student_r1shift-ckpt40-full (a separate directory, so the published 8-sample curve units at opd_student_r1shift-ckpt40/* are untouched), baseline student_init.

13.4 A2 — run-1 ckpt-100 at the FULL protocol: DROPPED

DROPPED by the user on 2026-08-26 (PREREGISTRATION E4) to keep condition D inside the $500 cap; run 1's ckpt-100 keeps its logged reduced-protocol asterisk. Its three roster rows read canceled, never pending.

13.5 Conditions B, C and D — CANCELED after termination collapse (training-only)

All three were launched on 2026-08-28 and all three collapsed. Their pre-registered first checkpoint save is at step 20, and all three crossed the collapse-watch threshold before it, so none of them published a policy that could be evaluated. They were cancelled at step ~22–25 by user decision (PREREGISTRATION §E5) and their GPU time was booked on the wall-clock basis. What follows is therefore training-only evidence: no accuracy number for B, C or D exists anywhere in this report, and none is estimated.

Per-step snapshots at steps 1 / 5 / 10 / 15 / 20, read from the babysitter's per-step tables (the only surviving record of a cancelled run — nothing was ever uploaded):

Grad-norm peaks and the KL controller. The two spikes that precede each collapse are the clearest early-warning signal in the whole study.

[image]

[image]

The two hypotheses these runs did settle, and they settled them by refutation.

• —H2 — UNTESTED (condition B): Collapsed at step 12; the first checkpoint save is at step 20, i.e. after the collapse, so no pre-collapse policy exists to evaluate. Recorded at the time: UNTESTED — the run never produced a pre-collapse checkpoint and was cancelled before any evaluation. H2 is carried by the B-short and B-lowlr redirects instead.
• —H3 — REFUTED (condition C): The KL coefficient was constant at 2.5 for all 24 recorded steps and the clip ratio still crossed 0.5 at step 12. Recorded at the time: REFUTED — the KL coefficient was 2.500 at every step and the run still collapsed, on the same step as B. A constant KL coefficient does not prevent termination collapse.
• —H4 — REFUTED (condition D): H4's premise was that the shift rewards the thinking student's native distribution (weightedrewardmean > 0 at step 1); the measured value is -0.01427, and the run collapsed at step 17 anyway. Recorded at the time: REFUTED — H4's own premise failed first: the step-1 weighted shift reward on the THINKING student was NEGATIVE, the same signature run 1 showed, and the run then collapsed like every other. The thinking student delayed the sink by ~5 steps; it did not avoid it.

Note what H3's refutation costs the fix list: the KL brake at its maximum does not prevent the sink. C held its coefficient at 2.5 at every single step — the driver's kl_controller clamps max(min, min(max, x)), and min == max pins it — and it collapsed on the same step as B, with a trajectory that matches B's to three decimal places for the first nine steps. A stronger anchor is not the answer, because the anchor was already at its strongest.

13.6 B-short — OpenThinker3 shift, 10 steps, save every 2

• —repo cmpatino/Qwen2.5-7B-Instruct-DirectOPD-OpenThinker3Shift-10steps @ 8b20748806f63d5b (pinned (training COMPLETE 2026-08-28, job 6a91a23c))
• —baseline student_init · teacher pair post_teacher_ot − pre_teacher_ot · lr 1e-6 · KL adaptive [0.5, 2.5], eps 0.01
• —Captures the policies B and C never saved. Trajectory reproduces B/C exactly (same seed): clip 0 through step 8, then response mean 474 -> 1,521 and clip 0.01 -> 0.29 at step 10.
• —Did ANY held-out capability move before the termination sink? ckpt-4 and ckpt-8 are clean pre-collapse policies (clip_ratio 0 through step 8); ckpt-10 is the onset step itself.

Training record.

10 of 10 pre-registered steps recorded (source: metrics.jsonl (model repo)). Collapse onset (first clip_ratio > 0.5): none; length-runaway marker (mean rollout length past 2x its step-1 value): step 10; peak clip ratio 0.289. Grad-norm peak 15.9 at step 9; KL coefficient 2.475 → 2.261; weighted shift reward -0.0354 → -0.0085, negative on 10 of 10 steps.

Note the two markers disagree, and the disagreement is the point: by the pre-registered collapse-watch rule (clip_ratio > 0.5) this run never collapsed, but the mean rollout length had already more than doubled by step 10 (peak clip 0.289). The onset signature is present; the run simply ended before the clip ratio caught up.

Held-out gains, per evaluated checkpoint (paired per-problem bootstrap vs student_init; the protocol is detected from each unit, never assumed)

Truncation at the 31,744-token evaluation cap, per checkpoint: step 4 aime24 1.9 %; step 4 aime25 1.0 %; step 4 math500 0.2 %; step 8 aime24 2.8 %; step 8 aime25 1.1 %; step 8 math500 0.8 %; step 10 aime24 44.2 %; step 10 aime25 41.2 %; step 10 math500 60.5 %.

⚠ Read the high-truncation rows as termination artifacts, not as capability being destroyed. step 10 on math500 is 60.5 % truncated and scores +0.50 pp. A policy that never emits a stop token produces a 31,744-token sample with no final answer in it, and the grader scores that as wrong however much the model knows — B-short's ckpt-10 is the control for exactly this: it is 41–60 % truncated and its accuracy is nonetheless preserved, because its answers still appear before the loop begins. The deeper into the sink a checkpoint is, the more of its samples never reach an answer at all.

Transfer ratio at the terminal checkpoint (student gain ÷ the OpenThinker3 pair's teacher gain, five guards)

H2 (pre-collapse form) verdict: NOT SUPPORTED

• —gain_on_at_least_one_benchmark — FAILED: no benchmark shows a positive paired gain whose 95 % CI excludes 0

13.7 B-lowlr — OpenThinker3 shift at OPTIM_LR 2e-7, 100 steps

• —repo cmpatino/Qwen2.5-7B-Instruct-DirectOPD-OpenThinker3Shift-lr2e-7-100 @ 6ad3add5b065a25c (resolved from `main` at build time (training COMPLETE 2026-08-28, job 6a919e1c, 100/100 steps))
• —baseline student_init · teacher pair post_teacher_ot − pre_teacher_ot · lr 2e-7 (logged DEVIATION from the pinned 1e-6) · KL adaptive [0.5, 2.5], eps 0.01
• —The 'can it be made to work' run: smaller steps so the policy can follow the gentle shift gradient without the grad-norm-spike-driven jump into the repetition sink (spikes of 26.5 and 21 preceded B's and C's collapses).
• —H5: with lr 2e-7 the run does NOT collapse (clip_ratio < 0.5 through step 100) AND ckpt-100 shows a paired gain on at least one held-out benchmark.

Training record.

100 of 100 pre-registered steps recorded (source: metrics.jsonl (model repo)). Collapse onset (first clip_ratio > 0.5): 48; length-runaway marker (mean rollout length past 2x its step-1 value): step 42; peak clip ratio 1.000. Grad-norm peak 49.8 at step 21; KL coefficient 2.475 → 0.915; weighted shift reward -0.0354 → -0.0038, negative on 100 of 100 steps.

Held-out gains, per evaluated checkpoint (paired per-problem bootstrap vs student_init; the protocol is detected from each unit, never assumed)

Truncation at the 31,744-token evaluation cap, per checkpoint: step 20 aime24 1.8 %; step 20 aime25 0.6 %; step 20 math500 0.4 %; step 40 aime24 3.1 %; step 40 aime25 0.6 %; step 40 math500 0.9 %; step 60 aime24 88.8 %; step 80 aime24 98.8 %; step 100 aime24 99.6 %; step 100 aime25 100.0 %; step 100 math500 100.0 %.

⚠ Read the high-truncation rows as termination artifacts, not as capability being destroyed. step 60 on aime24 is 88.8 % truncated and scores -4.58 pp; step 80 on aime24 is 98.8 % truncated and scores -2.50 pp; step 100 on aime24 is 99.6 % truncated and scores -2.50 pp; step 100 on aime25 is 100.0 % truncated and scores +1.25 pp; step 100 on math500 is 100.0 % truncated and scores -28.75 pp. A policy that never emits a stop token produces a 31,744-token sample with no final answer in it, and the grader scores that as wrong however much the model knows — B-short's ckpt-10 is the control for exactly this: it is 41–60 % truncated and its accuracy is nonetheless preserved, because its answers still appear before the loop begins. The deeper into the sink a checkpoint is, the more of its samples never reach an answer at all.

Only the (checkpoint × benchmark) cells that were approved for this condition appear above: step 60 on aime24; step 80 on aime24. The other combinations were never launched and are not counted as pending.

Transfer ratio at the terminal checkpoint (student gain ÷ the OpenThinker3 pair's teacher gain, five guards)

H5 verdict: NOT SUPPORTED

• —no_collapse_through_step_100 — FAILED: clip_ratio first exceeded 0.5 at step 48
• —gain_on_at_least_one_benchmark — FAILED: no benchmark shows a positive paired gain whose 95 % CI excludes 0

13.8 Cross-condition summary — one row per (condition, benchmark)

12 of 21 cells computed, 9 canceled. Each condition's student gain is divided by the teacher gain OF ITS OWN PAIR (run 1 -> r1distillsft; B/C/D and both redirects -> openthinker3sft). A pending cell means the unit is not on the Hub yet; a canceled cell means the unit can never exist because the run was stopped before it saved a checkpoint. Neither is ever an estimate. For B-short the row is its TERMINAL checkpoint (step 10); the per-step table is in the condition's own section.