galimova/mhc-mup-diagnostics

Dynamic mHC µP Diagnostic Report

Empirical Verification of Theorem Conditions for V4-style Dynamic Residual Routing

Date: 2026-01-05 Config: DeepSeek V4 (ns=4, K=20, hiddensize=7168, 61 layers)

Executive Summary

We ran all six diagnostics prescribed by the updated plan against a pure-PyTorch reimplementation of DeepSeek V4's dynamic mHC mechanism, sweeping width d ∈ {64, 128, 256, 512, 1024} with fixed n_s = 4 and K = 20.

The headline finding: Under standard initialization (σW = 0.02, sC = 0.1), the dynamic generator sensitivity ||DC(x)|| · ||x|| scales as d^0.98 ≈ d^1, violating the O(1) condition required by Theorem D (condition 4). This is the central technical obstacle for µP transfer in dynamic mHC.

The fix: Multiple corrected parameterizations restore O(1) scaling. The cleanest are:

• —Correction A (sC = 1/(ns·d)): yields d^{-0.035} ≈ O(1) for comb
• —Correction C (σW = 1/(ns·d)): yields d^{-0.037} ≈ O(1) for all three (comb, pre, post)

Correction C is the µP-natural choice: it corresponds to standard µP fan-in initialization for the generator weight W_C.

Diagnostic Results

Diagnostic 1: Finite-K Sinkhorn Error ε_K

Conclusion: εK ≈ eps = 1e-6 (the additive epsilon in the Sinkhorn loop dominates). With K=20 and ns=4, Sinkhorn convergence is essentially exact. The finite-K error is not a practical concern for V4's configuration.

Impact on Theorem A: The frozen realized comb stability theorem holds essentially exactly: ||C||2 = 1 + O(εK) with ε_K ≈ 10^{-6}.

Diagnostic 2: Frozen Mixer Spectral Norm ||C||_2

Conclusion: ||C||_2 ≈ 1 to 6 decimal places. The Birkhoff constraint works exactly as Theorem A predicts. The realized comb is nonexpansive.

Note: The high condition number κ(C) means that while the maximum singular value is 1, the minimum singular value is small (~0.001). This means C is close to a rank-deficient matrix — consistent with Sinkhorn producing near-permutation matrices when logits have moderate spread.

Diagnostic 3: Sinkhorn Quotient-Jacobian Spectrum

This diagnostic is width-independent (operates on fixed n_s=4 matrices).

Key Findings:

1. 1.The quotient Jacobian is well-conditioned. κ ≈ 7-10 across all K values. This validates Theorem C's assumption.

1. 1.σ_max ≈ 0.35, σ_min ≈ 0.06. The Sinkhorn projection is a contraction on G^⊥ (σ_max < 1). Perturbations are damped, not amplified.

1. 1.Gauge leakage drops exponentially with K. At K=20, leakage is 2e-6. The Sinkhorn Jacobian maps G^⊥ almost perfectly into G^⊥.

1. 1.The spectrum is K-independent for K ≥ 5. Convergence is fast.

1. 1.The (n_s-1)² = 9 singular values have a smooth distribution — all gauge-perpendicular directions are treated comparably.

Diagnostic 4: Dynamic Sensitivity (THE KEY RESULT)

Scaling exponents:

• —||DC(x)||·||x|| ~ d^{0.98}
• —||Dp(x)||·||x|| ~ d^{0.98}
• —||Dq(x)||·||x|| ~ d^{0.98}

This is the smoking gun. All three dynamic sensitivities scale linearly with width.

Jacobian Chain Decomposition

The two √d factors:

1. 1.||W_comb||₂ ~ √d: Generator weight spectral norm (fan-in = n_s·d)
2. 2.||x|| ~ √d: Multi-stream state norm (n_s·d entries)

Product: O(1) · O(1) · √d · O(1) · √d = Θ(d)

Diagnostic 5: Generated-Logit Update Scale

Conclusions:

1. 1.Almost all of ΔZ is in G^⊥ (>99.7%). Gradient updates naturally avoid gauge directions.
2. 2.||Π_{G^⊥} ΔZ||₂ grows with width (~√d), requiring LR compensation.

Diagnostic 6: Pre/Post Gate Statistics

Gates are stable across widths. Pre-weights center at 0.5 (sigmoid midpoint), post-weights at 1.0 (2·sigmoid(0)).

Corrected Parameterizations

The µP Rule for Dynamic mHC Generator Weights

Key insight: The generator weight's effective fan-in is n_s·d (the total multi-stream dimension), not d.

Implications for Theorem D

The conditions are:

1. 1.✅ Branch f_ℓ^a satisfies standard spectral µP (assumed)
2. 2.✅ εK = O(1), actually εK ≈ 10^{-6} (Diagnostic 1)
3. 3.✅ Quotient Jacobian well-conditioned, κ ≈ 7-10 (Diagnostic 3)
4. 4.✅→ Requires Correction C: σW = 1/(ns·d) gives O(1) (Diagnostic 4)
5. 5.✅→ Requires LR scaling: η_W = Θ(1/d) (Diagnostic 5)

With both corrections applied, all five conditions of Theorem D are satisfied.

V4 Sinkhorn Implementation Notes

From kernel.py:

1. 1.Init: Row-softmax + eps, then col-normalize
2. 2.Iterations: K-1 repetitions of (row-normalize, col-normalize)
3. 3.Convention: comb[j,k] with j=output stream, k=input stream
4. 4.hc_post: yo = qo · f(y) + Σi C[i,o] · residuali

Files

• —mhc_diagnostics.py — Complete diagnostic implementation (all 6 diagnostics)
• —mhc_analysis.py — Chain decomposition, corrections, figures