Modelpublic

hbfreed/Qwen3.6-35B-A3B-winnow-keep25-healed

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes

Model Card

Qwen3.6-35B-A3B · Winnow keep-25 (healed)

Qwen/Qwen3.6-35B-A3B with 75% of routed-expert channels removed by winnow variable-width expert pruning — 10.5B parameters. Requires trust_remote_code=True (ragged variable-width experts, PrunedQwen3_5MoeForCausalLM); transformers ≥ 5.

Instead of deleting whole experts (REAP-style), winnow shrinks each expert's SwiGLU intermediate width according to a per-channel importance score (router weight × |post-SwiGLU activation| × down-projection column L2), allocated by a per-layer global rank in blocks of 128 channels. Attention, embeddings and the router are untouched. Calibration: C4, qwen35_stats1024.

Layout

Raw training run — final weights are `checkpoints/step0200/` (load that subfolder). Intermediate checkpoints: step0050, step0075, step0100, step0125, step0175, step0200. trainer_state.pt files are optimizer state for resuming and are not needed for inference. heal_run/ holds args.json, the console log and per-step train_log.jsonl.

Healing recipe

On-policy distillation (scripts/62_qwen_distill_on_policy.py): reverse KL on the student's own rollouts against a bf16 Qwen3.6-35B-A3B teacher, 200 steps × 120k loss tokens, AdamW8bit, rollouts sampled with a 50/50 per-prompt mix of thinking / non-thinking chat frames (max 1024 new tokens), vLLM rollout engine refreshed every step, seed 1224. Trained on 3 × RTX Pro 6000 (Verda), 2026-07-22.

Results (step 0200, non-thinking greedy unless noted)

model	GSM8K (1319)	HumanEval pass@1 (pass@5)
Qwen3.6-35B-A3B (base)	0.957	0.940 (0.945)
keep-25 unhealed	—	—
keep-25 healed @200 (this)	0.768	0.059 (—)

Capacity ceiling: keep-25's lower training KL than keep-50 does not translate to downstream recovery. Evals via openbench (Protocol E), served through vLLM.