open-athena/densemixer-ab-qwen3-30b-a3b-thinking-opencode-serveparity-idEval

DenseMixer A/B — serve-parity ID-eval traces + weight-delta/routing analysis

Full artifacts for the controlled paired-init A/B ablation testing whether DenseMixer (training-only dense-forward + STE counterfactual router gradient; yaof20/DenseMixer, Axolotl integrations/densemixer/) improves MoE SFT quality — the empirical answer to marin-community/marin#7088.

Setup (identical except ONE flag)

• —Base / θ₀: Qwen/Qwen3-30B-A3B-Thinking-2507 @ 144afc2f (shared init).
• —Data: penfever/nemotron-code-oracle-filtered-qwen3.5-122b-131k-opencode-traces @ 6cc0c0b5 (serve-parity rebuild; <tools> system block + structured tool_calls + role:tool), pinned identical for both arms.
• —A — dense (DenseMixer ON): dense_mixer: true. HF model laion/qwen3-30b-a3b-thinking-opencode-sft-densemixer-serveparity.
• —B — sparse (control, DenseMixer OFF): dense_mixer: false. HF model laion/qwen3-30b-a3b-thinking-opencode-sft-sparse-serveparity.
• —Both: 366 steps, seed 42, same data + order, same LR schedule, ZeRO-3, seq 16384, chattemplate `tokenizerdefault` (train==serve).
• —Eval: agentic opencode on DCAgent/dev_set_v2 (100 tasks × 3 reps), TACC Vista, fair 64k config (max_input 65536 / max_output 16384).

Headline result — DenseMixer is functionally INERT

• —Sparse ≳ dense (Δ ≈ 3.7pp valid-only); difference NOT significant (z ≈ 1.66, p ≈ 0.10). DenseMixer gave no measurable ID-eval benefit (nominally slightly worse).
• —B (sparse) is PARTIAL — cancelled after a ~5h Daytona snapshot-cap stall at 22h wall — but it reached the same valid-trial count as A (263 vs 264, both ~88%), so the comparison is well-matched.
• —Both legs are BELOW the harvest gate (>10% infra-error: SandboxBuildFailedError 33 + DaytonaError 3 per leg; <90% complete) → not DB-registered; these are confirmatory of an already-established conclusion.

Weight-space + routing evidence (paired-init Δθ, all 18,867 tensors / 6,144 experts)

• —Routing decisions unchanged: top-8 overlap vs θ₀ = dense 0.859 ≈ sparse 0.860 (Δ < 0.001). DenseMixer's "correct router gradient" did not change which experts are selected relative to plain sparse SFT.
• —Outputs near-identical: dense-vs-sparse output-distribution divergence mean JS = 0.0035 (p50 0.0008, p90 0.0089) on a 49k-token probe.
• —Zero dead-expert revival: of 178 experts near-frozen under sparse, DenseMixer revived 0.
• —The only signature: the router Δθ points a different direction (cross-run cosine 0.435 vs 0.60–0.74 elsewhere) along a more low-rank set (dense stable-rank 13.3 vs sparse 22.2) — but this router re-steer produced no downstream routing or output consequence.

Conclusion: for this model/data/regime the DenseMixer weight-delta was a router-only low-rank re-steer that did not alter routing or outputs, and SFT showed no benefit. The A(complete) + B(partial) ID-eval confirms it.

Contents

• —densemixer_ab_A_dense_traces.tgz — full per-trial eval traces for leg A (dense), eval_jobs/<task>__<id>/ (result.json, config.json, agent/trajectory.json, verifier/, manifest.json).
• —densemixer_ab_B_sparse_traces.tgz — same for leg B (sparse).
• —analysis/ — routing_3A.json (per-layer top-k overlap / entropy), kl_2D.json (JS/KL), svd_summary.json, expert_revival.json, per_bucket_summary.csv (per-module Δθ magnitude/cosine).
• —results/ — the two aggregate result.json (+ A result_with_std_error.json), the eval config.json per leg, per_task_summary.csv (88 tasks, dense vs sparse per-task mean).

Small files are also mirrored in the companion gist linked from marin#7088.