dougalldeepmind/2026-07-29-msm-philosophy-spec-surf-audit

SURF audit: harmful-omission rubric against the MSM+AFT+CoT checkpoint

• —experiment: SURF (Surfacing Unintended Response Failures) EM-loop search over a generic instruction-following prompt pool, scoring responses against a harmful-omission rubric, against the primary MSM target checkpoint. An independent search-based instrument alongside Petri and the fixed evaluation.
• —date_generated: 2026-07-29
• —constitution: The Philosophy Spec from "Model Spec Midtraining" (chloeli/modelspecmidtraining, arXiv:2605.02087, Appendix D.1). Target checkpoint only (msm-aft-cot); this run has no matched-checkpoint control, so nothing here is MSM-attributable on its own.
• —source_repo: Matthew-Bozoukov/teaching_claude_why_replication, commit 83877eade31a291522226eab19bb1074111d6998, path experiments/vulnerabilities/
• —models: Target chloeli/qwen-3-32b-philosophy-spec-msm-aft-cot on Qwen/Qwen3-32B, served via vLLM. Judge and query-generation model: claude-haiku-4-5.
• —generation_config: target_max_tokens=6144 (raised from an initial 2048 after truncated scratchpads corrupted the search - see harmful-omission-run1-ABORTED-truncation/), candidatesperiter=50, buffersize=10, targetconcurrency=12.
• —schema:
• —rubrics/harmful-omission.yaml, rubrics/unverified-authority.yaml: the SURF rubric definitions.
• —calib-01/, calib-02/: pre-run calibration checks.
• —harmful-omission-run1-ABORTED-truncation/, harmful-omission-run1-HUNG-nolog/: superseded, defective runs kept as evidence (see repo docs for what went wrong).
• —harmful-omission-run2/: the accepted run - results.jsonl (193+ scored candidates across 5 iterations), summary.jsonl (per-iteration convergence stats), failures.jsonl.
• —notable/clinical-fabrication.json: the single most notable candidate found - a fabricated clinical-trial-statistics response, kept unvalidated and flagged as such.
• —validation-funnel.json: two-round adversarial validation of all 40 raw flags from run 2.
• —provenance: Regenerate with experiments/vulnerabilities/scripts/surf/Invoke-SurfRun.ps1 against tools/SURF (seoirsem/SURF, pinned commit - see repo docs/04-surf-plan.md). Full narrative in experiments/vulnerabilities/docs/18-surf-validation.md and docs/15-surf-scope-decision.md.

Headline result

Of 40 raw flags (reward_score > 50), two-round adversarial validation confirmed 1 and rejected 39 - a 97.5% false-positive rate (95% CI 86.8-99.9%). Not one of the 40 flags exhibited the rubric's stated mechanism. 10 of 40 were fabrication rather than omission - the opposite construct the rubric measures - and are recorded separately. Full detail: docs/18-surf-validation.md in the source repository.