arianraje/qwen3-4b-gdn-hybrid-stage1-align-dtfix

qwen3-4b-gdn-hybrid-stage1-align-dtfix

Stage-1 endpoint (dt_bias fixed) — per-layer hidden-state alignment, 100M tokens @4k, LR 3e-3 (identical recipe to qwen3-4b-gdn-hybrid-stage1-align). Val rel-MSE 0.0041 mean (original 0.0087; layer-16 outlier 0.126 -> 0.011), wikitext-2 PPL 34.1 (original 29.5). 0-shot likelihood battery vs the original stage-1 on the same harness: PIQA 75.5 vs 63.8, HellaSwag 58.0 vs 56.5, ARC-Easy 75.6 vs 48.6, ARC-Challenge 49.2 vs 32.6, Winogrande 59.2 vs 60.6 (likelihoodbatterycmu.json; teacher 74.9/68.5/78.5/53.8/65.8).

Part of a study converting full-attention Qwen3-4B into a GDN (gated DeltaNet) hybrid (27 of 36 layers converted, uniform 1:4 retention) and recovering capability via staged distillation. Checkpoints are stock Qwen3NextForCausalLM — load with AutoModelForCausalLM (transformers >= 4.57).

`-dtfix` series. transformers' Qwen3Next port initializes the GDN time-step bias with dt_bias.fill_(1.0); with A_log = log U(0,16) and the decay g = -A * softplus(a + dt_bias) that gives retention exp(-dt*A) < 1e-3 for 2/3 of heads at init — the recurrent state starts dead and the gradient into dt_bias/A_log is too small to recover it (the original qwen3-4b-gdn-hybrid-* series carries this through every stage). This series re-runs the pipeline from a surgery that uses the fla GatedDeltaNet reference init (dt ~ LogUniform(1e-3, 0.1), dt_bias = softplus^-1(dt); median retention 0.94). Everything else (inheritance map, data recipe, trainer, LRs, budgets) is unchanged.