experiment | Arm: REPLACE each of the 702 principle-scoped difficult-advice rows' reasoning traces with the empty think marker, leaving prompt and answer byte-identical. The marker is masked whole by the existing generation-boundary rule, so the model is supervised on the visible answer and never learns to emit an empty close. Tests whether the reasoning was doing work AS CONTEXT: this arm and the answer-only arm supervise the same answer tokens and differ only in whether the trace is present to condition on. |
date_generated | 2026-09-01 |
constitution | constitutions/claudedistilled07principlesapproved/constitution.md (via the training data; never quoted in the trained text) |
source_repo | teachingclaudewhy_replication @ 7137baea6e1f33c182908af6bfba29634d87c9e8 |
models | token stream Qwen/Qwen3.6-27B (tokenizer + ModelProfile literals) |
generation_config | none - no model is sampled here. The build is a deterministic text rewrite over a pinned control mixture. |
schema | text (rendered Qwen3.6 chat; the 702 difficult-advice rows have their think block replaced by the empty marker, prompt and answer unchanged; the 9,284 Table2 rows are byte-identical to the control); source. NO supervise column: the default 'all' already masks the whole empty marker and supervises the answer, so no new mask mode was needed. |
provenance | uv run python scratch/emptycot/buildmixture.py ; every rewritten row asserted to keep its prompt and answer byte-for-byte, to open with the exact empty marker, and the census asserted at 0 real / all empty / 0 absent |
control_dataset | LASR-Callum/2026-08-21-table2-9284-difficult-advice-principle-scoped-702-train-mixture (t29284dachunkonly_702.jsonl @ fa98fadeee72) - same prompts and answers, real traces intact |
trained_control | LASR-Callum/2026-08-21-qwen36-lora-table2-9284-difficult-advice-chunk-only-702-rank-64-dynbatch - ODCV 11.5% [6.2, 19.6], severity 0.62 (2 passes). |
sibling_arms | cot-only (LASR-Callum/2026-08-31-cot-only-supervision-chunk-only-702, ODCV 9.5% [3.2, 17.5]) and answer-only (LASR-Callum/2026-09-01-answer-only-supervision-chunk-only-702). Those two partition the control's supervision exactly; THIS arm instead changes the context, supervising the same answer tokens as answer-only (702 fewer - the blank-line separator, which is part of the forced marker here). |
rows | 9986 (702 traces blanked) |
census | 0 real / 10348 empty / 0 absent assistant turns |
intervention_scale | at maxseqlen 8192: forward tokens 6,033,864 -> 5,614,529 (-6.95% overall, -41.01% on the difficult-advice rows); supervised tokens 2,867,910 -> 2,447,171 (-14.67%); difficult-advice share of the training signal 28.6% -> 16.4% |
caveat | This mixture carries ZERO real reasoning traces, which trips checkthinkingdeclaration's gotcha-2 guard; the train config waives it explicitly with allow_no_reasoning: true and the adapter records the waiver. The guard's actual failure -- training the model to EMIT an empty close -- cannot occur here, because the marker is masked whole and never earns loss. KNOWN MISMATCH: the arm is served and evaluated in THINKING mode for comparability with its siblings, so at inference the model generates a real trace it was never trained to condition on. |
