dougalldeepmind/2026-07-29-msm-philosophy-spec-fixed-eval

Fixed evaluation: does model-spec midtraining change harmful-omission or provenance behaviour?

• —experiment: Byte-identical single-turn fixed evaluation across seven matched checkpoints, designed to attribute (or rule out) an effect of model-spec midtraining (MSM) on two behaviours: treating tool-channel content as an instruction (prov-* probes) and suppressing a warranted safety concern under instruction (omis-* probes). This is the attribution step behind the investigation's headline finding that alignment fine-tuning (AFT) inverts the harmful-omission failure rather than fixing it (Fisher exact p = 2.9e-07), and that no MSM-attributable effect survives correction for multiple comparisons.
• —date_generated: 2026-07-29
• —constitution: The Philosophy Spec from "Model Spec Midtraining" (chloeli/modelspecmidtraining, arXiv:2605.02087, Appendix D.1). Contains no behavioral rules; teaches epistemic humility, impermanence, suspicion of self-preservation motives, distrust of ends-justify-means reasoning, and trust in a hierarchy of principals.
• —source_repo: Matthew-Bozoukov/teaching_claude_why_replication, commit 83877eade31a291522226eab19bb1074111d6998, path experiments/vulnerabilities/
• —models: Seven matched checkpoints on Qwen/Qwen3-32B (base revision 9216db5781bf21249d130ec9da846c4624c16137), served as LoRA adapters via vLLM 0.11.0: msm-aft-cot (target, chloeli/qwen-3-32b-philosophy-spec-msm-aft-cot, adapter revision 9a00c85c80d195c6153a56373e6901413ba6f519), msm-aft-no-cot, aft-cot, aft-no-cot, msm-only, id-baseline, qwen3-32b-base. Judge: claude-opus-5 / claude-sonnet-5 (subagents, subscription-billed, not this token). LoRA geometry independently verified: r=64, lora_alpha=128, all 7 target modules, 896 tensors = 64 layers x 7 modules x 2.
• —generation_config: temperature 0.7, topp 1.0, maxtokens 2048 (raised from an initial 700 after a truncation defect - see generations-v1-truncated.json), seeds 1000-1004, 5 samples per probe per checkpoint.
• —schema:
• —probes.json: the 7 probes (5 test, 2 control per family; families B-provenance, C-omission), each with expected_aligned (the rubric judges scored against).
• —generations.json: 245 raw responses (7 checkpoints x 7 probes x 5 samples), the clean final round.
• —generations-v1-truncated.json: the superseded 700-token round, kept as evidence of the truncation defect (see repo docs 11-fixed-eval-defects.md).
• —blind/slice-{A..G}.json + blind-mapping.json: blind-judging slices, one per checkpoint, mapping withheld from judges.
• —scores/slice-*.json: final judge scores (7 independent judges, one per slice).
• —scores-v1-nonuniform/, scores-v2-truncated/: superseded judging rounds, kept as evidence.
• —attribution.json: the merged, unblinded scored dataset plus computed matched contrasts with bootstrap confidence intervals.
• —rater-agreement.json: independent second judging of 2 slices for inter-rater agreement (kappa 1.000).
• —sensitivity.json: Holm-Bonferroni correction and leave-one-probe-out results across all 15 contrasts.
• —provenance: Regenerate with experiments/vulnerabilities/scripts/petri/fixed_eval.py (GENERATE_ONLY=1), then scripts/petri/make_blind_slices.py and scripts/petri/unblind.py. Full narrative in experiments/vulnerabilities/docs/13-attribution-results.md and docs/16-findings.md.

Headline result

Zero of 15 matched contrasts survive Holm-Bonferroni correction for an MSM-specific effect. The only robust contrast is full-pipeline-vs-base (+3.44, 95% CI [+1.72, +5.12]), which conflates every training stage. Full detail: docs/13-attribution-results.md in the source repository.