ThakiCloud/Qwen3.8-27B-W4A16-GPTQ
Qwen3.8-27B-W4A16-GPTQ
18.59 GB. The smallest checkpoint we have of this model, and on the benchmarks we ran it is not distinguishable from bf16. It exists because a 30 GB NVFP4 build does not fit a 32 GB card and this one does, with room left for KV cache.
We also publish the thing most quantization cards leave out: we measured how far two builds of this exact recipe land from each other, and on one axis it is very far. Read the noise-floor section before you compare this checkpoint against anything.
What it is
GPTQ W4A16 via llmcompressor, pack-quantized, group size 128, symmetric int4, actorder: static, dampening_frac: 0.01. Calibration: 1,024 samples at 2,048 tokens. Quantization took ~38 minutes on a single GPU.
Ignored (left at original precision): re:.*vision.*, re:.*visual.*, lm_head, re:.*embed_tokens.*. This is a hybrid VL model — 64 layers, 16 full-attention and 48 linear-attention (GDN) — so the naive "27B at 4-bit ≈ 13.5 GB" arithmetic does not hold. The vision tower, embeddings and recurrent state stay wide, which is why the file is 18.59 GB rather than ~14.
Quality
GSM8K, full 1,319-item set, greedy, vLLM 0.27.1 + lm-eval 0.4.12. All arms in one serving session, identical serve config, model path the only difference.
Against bf16 on flexible-extract this build is +2.28pp (en) and +1.52pp (ko) — that is, slightly ahead, and well inside noise either way. Against a third-party INT4 build of the same model it is +0.76pp / +0.08pp. On these tasks the three are not distinguishable.
⛔ Judge on `flexible-extract`, not `strict-match`. strict-match requires the answer wrapped in a specific format; 4-bit builds of this model (ours and third-party alike) often answer correctly without that wrapper and get scored wrong. We had four blind judges read 40 disputed cases: 35 of 40 were correct answers rejected on formatting.
⛔ Rebuild noise floor — read this before comparing checkpoints
We ran the same recipe a second time — same model, same settings, same calibration — and evaluated both builds in one session. They did not land in the same place:
The last row is the one that matters. On questions whose answer is not in the retrieved document, this build fabricated an answer 9.17% of the time (n=240). The second build of the same recipe did so 36.25% of the time. A third-party INT4 build scored 1.25%; bf16 scored 3.33%.
Two consequences, stated plainly:
- A difference under 2× the floor for that metric is not attributable to the recipe. That rule invalidates several comparisons we ourselves had drawn, including some in our sibling cards.
- Our GPTQ pipeline is not reproducible on abstention behaviour. Run it twice and you cannot predict whether you get 9% or 36%. We do not currently know why. If refusing to answer when the document is silent is load-bearing for you, use a frozen third-party artifact instead of rebuilding this recipe — or evaluate the specific file you intend to ship.
And this failure is invisible to accuracy metrics: the 36% build had the highest extraction accuracy of the four arms and an identical grounded-answer score. An accuracy-only gate ships it.
Full data: `ThakiCloud/quantization-rebuild-noise-floor`.
Throughput — this is the slow one, and that is the trade
Same run, one B200, identical serve config, 2,048 in / 256 out:
3.52× slower at the ceiling and 5.1× worse TTFT. W4A16 goes through Marlin dequantization; NVFP4 runs a native Blackwell kernel. So the choice is not "which is better" but which constraint binds:
- 32 GB card → this build. NVFP4's 30.14 GB leaves no room for KV cache.
- 96 GB card → take the NVFP4 build; the memory you saved buys nothing there.
On long context the ordering holds. KV on this model is 64 KiB/token (only the 16 full-attention layers contribute), and --kv-cache-dtype fp8 halves that for 1.95× the KV pool at no measured throughput or quality cost — worth turning on before you shop for a bigger card.
Usage
vllm serve ThakiCloud/Qwen3.8-27B-W4A16-GPTQ \
--max-model-len 131072 --max-num-seqs 256 \
--kv-cache-dtype fp8 --mamba-cache-mode align--mamba-cache-mode align is required — this is a hybrid SSM model. Native context is 262,144.
Scope of what we measured
GSM8K (en/ko, full set), a Korean procurement-document RAG suite (extraction / grounded / absent, n=240), throughput and TTFT on one B200. One model, one method, one hardware generation. We did not measure: MMLU-class breadth, code, long-context retrieval quality, or behaviour on other GPU families.