dougalldeepmind/2026-07-31-qwen36-27b-mmlu-capability-eval

2026-07-31 — MMLU capability eval: Qwen3.6-27B constitution-SFT arm ladder

• —experiment: Absolute-benchmark (MMLU) capability check that mixing synthetic constitution / difficult-advice documents into a Tulu SFT mixture does not cost Qwen3.6-27B general knowledge — the guardrail under the alignment result, run across the full mixture-ratio arm ladder against the untuned base model.
• —date_generated: 2026-07-31 (think/, primary) and 2026-07-30 (nothink/, companion run)
• —constitution: The distilled Claude constitution / difficult-advice corpus this project trains on (see constitutions/ in the source repo and the dataset matboz/difficult-advice-qwen3). Arms B/C/D are LoRA SFT on 10/20/40% synthetic constitution-document mixtures with Tulu; arm A is 100% Tulu (no constitution data).
• —source_repo: https://github.com/Matthew-Bozoukov/teachingclaudewhyreplication — harness at commit `51117d1` ("lmsys + mmlu + proxy evals": `src/mmlu.py`, `src/experiments/mmlueval.py, src/experiments/mmlureport.py`, `configs/mmlueval.yaml, configs/capabilityeval.yaml`). Exact resolved config for every arm is embedded in each `runmeta.json` beside the records.
• —models:
• —base: Qwen/Qwen3.6-27B (served via vLLM 0.26, one process serving base + all adapters as LoRA modules — identical build/flags for every arm)
• —LASR-Callum/2026-07-30-qwen36-tulu-100-pct-lora (armasynth00 — unmatched control: 2 epochs / packing on, vs 1 epoch / packing off for B/C/D)
• —LASR-Callum/2026-07-29-qwen36-difficult-advice-tulu-lora-10-90 (armbsynth10)
• —LASR-Callum/2026-07-28-qwen36-difficult-advice-tulu-lora-20-80 (armcsynth20)
• —LASR-Callum/2026-07-29-qwen36-difficult-advice-tulu-lora-40-60 (armdsynth40)
• —generation_config: temperature 0.0, topp 1.0, one sample, seed 0, 5-shot (dev split of the same subject), choices shuffled per question (seeded by uid). `think/`: `enablethinking=true; token budget 4096, with truncated (finishreason=length`) records re-generated at 7168 and finally 15000 tokens against a 16384-token window — safe to mix at temperature 0 because an answer that stopped naturally is identical under a larger cap. `nothink/`: `enablethinking=false`, max_tokens 2048.
• —schema: <mode>/<arm>/records.jsonl, one row per question: uid (subject/index into cais/mmlu test), prompt_hash, raw (full generation), think (reasoning trace), answer (visible answer), finish_reason, parsed (extracted letter or null), parse_tier, correct, answer_letter (gold), think_words. Per-arm <ts>/metrics.json carries accuracy, Wilson CI, parserate, truncationrate; report_think/ and report_nothink/ carry the paired comparison (mmlu_results.md, mmlu_scores.json, figures).
• —provenance: uv run python src/experiments/mmlu_eval.py --arms all --endpoint <vllm>/v1 --parallel 32 then uv run python src/experiments/mmlu_report.py (add --nothink for the companion run), at commit 51117d1. Subset: 10 questions/subject × 57 subjects = 570, stratified, seed 0, subset hash 3952064292260029 (identical for all arms; paired statistics). Server: scripts/runpod_capability.py up (RunPod H100, vLLM --enforce-eager --max-model-len 16384).

Headline result (think mode, n=570 paired)

*armd's CI lower bound (−3.2pp) sits 0.2pp below the −3pp margin with a point estimate of −1.1pp: an underpowered interval at n=570, not a demonstrated regression. Do not report as a regression; `--persubject 20` tightens it.

Flat dose-response across 10→20→40% synthetic: constitution-document SFT costs no measurable MMLU capability. Base-arm caveat: 4 of 570 questions still ruminate past a 15k-token thinking budget and score wrong (0.7% truncation, within the 2% gate). The nothink companion run shows the same flat picture (base 87.0%, arms within 1.6pp; arm A's raw −7.5pp there is a parse-format artifact — its parsed-only accuracy equals base — diagnosed in report_nothink/).