experiment | Ablation arm built in two passes over BOTH halves of every difficult-advice row. Pass 1 rewrites the reasoning and the answer to drop four deliberative moves: engaging the tempting option, drawing an analytic distinction, enumerating outcome branches, and offering an alternative route. Pass 2 then labels every remaining unit and DELETES the ones that still offer an alternative. A SECOND deletion pass over the answers then removes units offering an alternative OR engaging the tempting option. Neither method works alone: rewriting left the alternative in 34% of answers because the model would not fully comply, and deleting alone left it in 40% because the offering is diffuse. Rewrite plus two deletion passes takes it to 3.6%. |
date_generated | 2026-08-22 |
constitution | claudedistilled12principlesmid (constitutions/claudedistilled12principlesmid/constitution.md) |
source_repo | Matthew-Bozoukov/teachingclaudewhy_replication @ 7acd0ff0abdd2f38846b5353c22f45096f0f7608 |
models | rewriter anthropic/claude-sonnet-5; unit labeller and blind rater google/gemini-3-flash-preview |
generation_config | rewriter temperature 0.7, maxtokens 8192; labeller and rater temperature 0.0 with JSON responseformat; rated items shuffled and shown one per call with no version label |
schema | text (rendered Qwen3.6 chat); source; scenarioid and bothrewritten present only on the 711 rewritten rows; mixturestats.json is the dashboard-shaped sidecar ({total.examples, bysource}) counted from the rows file at copy time, beside the pipeline's original mixturestatsablated.json |
provenance | scratch/runrewritepipeline.py --stage rewrite, then --stage ablate, then scratch/ablatealternative.py --half answer --moves offersalternative,engagescounterargument --tag pass2, then scratch/rateanswerresponsiveness.py, then scratch/buildfinalmixture.py. Sample manifest: output/rewritecache/sids.json |
control_dataset | LASR-Callum/2026-08-06-table2-9284-synthdoc-716-train (mixture_think.jsonl) — the unmodified 10,000 rows. Siblings: matboz/2026-08-21-ruleform-t2-9284-synthdoc-716 (reasoning only) and matboz/2026-08-22-ruleform-both-t2-9284-synthdoc-716 (both halves, rewrite only). |
rows | 9,960 rows: 9,284 Table2 rows byte-identical to the control, plus 676 difficult-advice rows with both halves rewritten and ablated twice. FORTY rows were DROPPED: 16 whose answer the deletion emptied, and 24 that a blind responsiveness rater judged no longer to answer the question they were written for. |
verification | system prompt and user turn asserted byte-identical on all rows; the pre-swap answer asserted equal to the pipeline's own record; exactly 676 rows differ; all assistant turns carry think markers (676 real traces, 9,646 whole empty markers, 0 missing). 662/676 rows rated coherent in both halves after deletion. |
ablation_strength | Blind rater over all 676 rows, AFTER both passes: engagescounterargument reasoning 3% / answer 5%; analyticdistinction reasoning 29% / answer 52%; outcomebranches reasoning 1% / answer 4%; offersalternative reasoning 1% / answer 4%. All four targets absent in 475/676 reasoning traces and 312/676 answers. |
caveat | THE ANALYTIC DISTINCTION DID NOT LAND, and it is the whole reason this arm is not a clean test. It survives in 29% of reasoning traces and 52% of answers despite being an explicit target of the rewrite; it resisted a lexical-ban pass, a span-quoting repair loop, and a prompt-stripping probe. Three of four targets are essentially gone; read any result as 'engagement, branching and the alternative removed, distinction intact'. SECOND CONFOUND: length. Reasoning falls to ~34% of the control and answers to ~22%, and untargeted moves fell with them (returnsagency in answers 81% -> 15%, namestension 94% -> 10%). A downstream difference could be terseness rather than the four moves. THIRD CONFOUND, introduced by the filtering itself: the dropped rows are not spread evenly. Twenty of the forty are trait 8 ("be genuinely, substantively helpful -- never treat unhelpfulness as safe"), leaving 59 rows of that trait against ~79 for the others, because in those scenarios the alternative and the engagement WERE the answer. The corpus is therefore thinner on the trait most opposed to the ablation, and that imbalance is itself a candidate explanation for any downstream effect. |
copied_from | matboz/2026-08-22-ruleform-ablated2-t2-9284-synthdoc-676 @ f7751e71a6a0b7ebf061ca856662afbc2359642d — verbatim copy of the rows and the pipeline's own stats sidecar; nothing regenerated. |
