ThakiCloud/Qwen3-Coder-30B-A3B-Prune3-W4A16

Qwen3-Coder-30B-A3B-Prune3-W4A16

3.1% expert pruning (124 of 128 experts kept per layer) plus W4A16 quantization of Qwen3-Coder-30B-A3B-Instruct. 61.1 GB → 16.2 GB (3.76×), with no detectable loss on HumanEval.

Its sibling Qwen3-Coder-30B-A3B-W4A16 is the same recipe with no pruning, at 16.69 GB. The 3.1% prune buys 0.46 GB — about 2.8% — on top of what quantization already gave. We are publishing both so that number is visible, because the interesting part of this release is not either checkpoint. It is where the pruning stops being free.

Measured results

Single B200, vLLM, lm-eval, 150 questions per task, humaneval_instruct via local-chat-completions with --gen_kwargs continue_final_message=True,add_generation_prompt=False.

stderr is roughly 2 points at 150 questions, so 0.9467 is not better than the 0.9267 base — it is no detectable loss. GSM8K sits slightly low (0.8867 against a base in the 0.93–0.94 range); it is within or near noise but consistently on the low side, and we report it rather than drop it.

The cliff — why 3.1% and not more

Expert pruning on this model degrades as a steep slope, not a gentle curve. Swept in bf16 with no quantization involved, so the numbers isolate pruning alone:

The sharpest segment is between 7.8% and 8.6%: 0.8 percentage points more pruning costs 25 points of pass@1. Below about 6% the loss is modest; past about 8.6% it is catastrophic. 3.1% is the operating point that survives it.

An earlier internal checkpoint at 10.16% scored 0.28 on HumanEval — and its IFEval (0.79) and GSM8K (0.93) looked fine. Instruction-following and grade-school math apparently route through a redundant subset of experts; code generation does not. If you prune a MoE and only watch general benchmarks, this failure is invisible.

What this costs you: throughput

Nothing, relative to the unpruned 4-bit sibling — and that is the point. Single B200, vLLM, random dataset, input 512 / output 256, --ignore-eos:

Size note (added 2026-08-14). The FP8 baseline is recorded here as 29 GB from the 2026-08-06 run. The FP8 checkpoint currently in our registry measures 31.20 GB on disk. We have not confirmed the two are the same build, so both figures are left standing rather than one being quietly overwritten. Treat the FP8 size as approximate; the throughput ratios were measured against whichever build ran that day.

Two readings. First, the two prune ratios are throughput-identical, so pruning itself contributes essentially nothing to serving speed — what costs speed is the W4A16 format, whose 4-bit weights are dequantized at compute time while FP8 has native Blackwell tensor-core support. Second, FP8 leads by 1.27×–1.43× at every concurrency measured and holds better tail latency at concurrency 50 (322 ms vs 475 ms p99).

This is a capacity tool, not a throughput tool. Use it to fit a 61 GB model onto smaller or fewer GPUs. If it already fits and you want speed, use FP8, unless your hardware is Blackwell, in which case read on.

Speed versus NVFP4: this question is now answered

Earlier versions of this card said we had not tuned the W4A16 serving path and that an optimized kernel might narrow the gap to FP8. That hedge is now resolved, and not in W4A16's favor. A four-way comparison of bf16, FP8, W4A16, and NVFP4 — all on the same base model, same single B200, same workload (input 1746 tokens, output 256, vLLM 0.27.1, measured 2026-08-13) — found that the gap to FP8 does not close by tuning the W4A16 serving path. Kernel selection was not the missing piece.

It closes by switching quantization formats. NVFP4 quantizes activations as well as weights and runs directly on the Blackwell FP4 tensor cores, while W4A16 stores 4-bit weights and must dequantize them at compute time before every matmul. That is a structural difference between the two formats, not a tuning gap in this checkpoint or in the pruning applied to it.

The four-way ledger was run against the unpruned W4A16 checkpoint, not this pruned one, but as noted above the two are throughput-identical in our measurements — pruning at this level does not move serving speed, only the quantization format does. NVFP4 beats FP8 at every concurrency measured in the full sweep (1.07x to 1.32x) and beats W4A16 by 1.40x to 1.83x. The full sweep, tokens-per-joule numbers, and a HumanEval comparison across all four formats are on the NVFP4 sibling card: Qwen3-Coder-30B-A3B-NVFP4.

That does not make this checkpoint obsolete. This pruned-and-quantized checkpoint still wins on two things NVFP4 does not touch: it is the smallest of the family (16.2 GB against NVFP4's 18.11 GB), and it runs on Hopper (H200, H100), where NVFP4 has no native tensor-core path and falls back to Marlin emulation, measured at 0.85x of the equivalent FP8 throughput on that hardware. If you need the smallest possible footprint, or your target is Hopper and NVFP4 is therefore not an option, this is the format to use. It is not a speed recommendation: see the Hopper note below. If your target is Blackwell and you want speed as well as a smaller footprint than bf16 or FP8, use the NVFP4 sibling instead.

On Hopper it is still a memory play (added 2026-08-15)

We had inferred the Hopper case rather than measured it. It is measured now, on a single H200 NVL with the unpruned sibling of this checkpoint (input 1746 / output 256, vLLM 0.27.1), and 4-bit weight-only storage loses to FP8 at every concurrency there too:

The picture is therefore consistent across both GPU generations: this format buys memory, never throughput. On Blackwell the penalty is worse and grows with load (0.76x falling to 0.67x across the same band), because the faster the low-precision tensor cores are, the further a dequantize-before-you-compute path falls behind.

Measured on the unpruned sibling; pruning is orthogonal to the format's kernel path and we did not re-sweep this checkpoint separately. Ledger: 2026-08-15-per-hardware-quant-matrix.json.

Caveats

• —One model family, one code benchmark, 150 questions per task (stderr ~1.8–4 pp).
• —The FP8 baseline is 29 GB against 16.2 GB — a speed comparison, not an equal-memory one.
• —No attempt was made to tune the W4A16 serving path (e.g. Marlin kernel selection); an optimized 4-bit kernel could narrow the gap.
• —Router-KD recovery was tested and found inapplicable to removed-expert damage, so there is no recovery step in this recipe.

Provenance

Pruned and quantized by ThakiCloud with llm-compressor. Base model Qwen/Qwen3-Coder-30B-A3B-Instruct is Apache-2.0 and not gated; this derivative inherits that license. Prune budget and quantization recipe are in recipe.yaml.