Modelpublic

hbfreed/Qwen3.6-35B-A3B-winnow-keep25

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

1likes133downloads

Model Card

Qwen3.6-35B-A3B · Winnow keep-25

Qwen/Qwen3.6-35B-A3B with 75% of routed-expert channels removed by winnow variable-width expert pruning — 10.5B parameters. Requires trust_remote_code=True (ragged variable-width experts, PrunedQwen3_5MoeForCausalLM); transformers ≥ 5.

Instead of deleting whole experts (REAP-style), winnow shrinks each expert's SwiGLU intermediate width according to a per-channel importance score (router weight × |post-SwiGLU activation| × down-projection column L2), allocated by a per-layer global rank in blocks of 128 channels. Attention, embeddings and the router are untouched. Calibration: C4, qwen35_stats1024.

Results (unhealed, this checkpoint)


Keep fraction	0.25
Params	10.5B (base 35B)
C4 ppl, 32 × 2048-token seqs (`winnow_verify.json`)	22.53

Whole-expert REAP comparison at matched budget and calibration, and downstream numbers (MMLU-Pro / IFEval, Protocol E), are in the Winnow writeup. Healed variants: keep50-healed, keep25-healed.

Serving

vLLM via the winnow plugin in variable-reap; --enforce-eager, pipeline-parallel supported (keep-50 runs on 2 × 24 GB).