antonyMox/Qwen3.6-27B-AutoRound-INT4-wide-MTP

<div style="background:#0b1020; border:1px solid #1e293b; border-top:4px solid #22d3ee; border-radius:14px; padding:26px 28px; margin-bottom:22px; font-family:ui-monospace,SFMono-Regular,Menlo,monospace;"> <div style="color:#e2e8f0; font-size:12px; letter-spacing:2px;">// ANTONYMOX · PARK RECORD · SMARTEST BUILD</div> <div style="color:#ffffff; font-size:27px; font-weight:800; line-height:1.2; margin:8px 0 6px; font-family:-apple-system,Segoe UI,Roboto,sans-serif;">Qwen3.6-27B <span style="color:#22d3ee;">wide</span> — INT4 · MTP</div> <div style="color:#94a3b8; font-size:13.5px; line-height:1.6; max-width:660px; font-family:-apple-system,Segoe UI,Roboto,sans-serif;">The clean Qwen3.6-27B base, quantized our way — big linear-attention projections kept in BF16. Highest hard-coding score in our fleet (<b style="color:#e2e8f0;">78</b>), MTP head preserved, sees images, up to 256k context (native).</div> <div style="margin-top:15px; display:flex; flex-wrap:wrap; gap:7px;"> <span style="color:#67e8f9; border:1px solid #334155; border-radius:6px; padding:3px 9px; font-size:11px;">hard&nbsp;78&nbsp;· record</span> <span style="color:#86efac; border:1px solid #334155; border-radius:6px; padding:3px 9px; font-size:11px;">MTP&nbsp;preserved</span> <span style="color:#f9a8d4; border:1px solid #334155; border-radius:6px; padding:3px 9px; font-size:11px;">vision</span> <span style="color:#7dd3fc; border:1px solid #334155; border-radius:6px; padding:3px 9px; font-size:11px;">256k&nbsp;ctx</span> <span style="color:#fdba74; border:1px solid #334155; border-radius:6px; padding:3px 9px; font-size:11px;">26&nbsp;GB</span> <span style="color:#e2e8f0; border:1px solid #334155; border-radius:6px; padding:3px 9px; font-size:11px;">Apache-2.0</span> </div> </div>

<div style="display:flex; flex-wrap:wrap; background:#ffffff; border:1px solid #e2e8f0; border-radius:12px; overflow:hidden; margin-bottom:6px; font-family:-apple-system,Segoe UI,Roboto,sans-serif;"> <div style="flex:1; min-width:110px; padding:16px 10px; text-align:center; border-right:1px solid #eef2f7;"> <div style="font-size:26px; font-weight:800; color:#0891b2;">78</div> <div style="font-size:10.5px; color:#64748b; letter-spacing:0.8px;">LCB-HARD 🏆</div> </div> <div style="flex:1; min-width:110px; padding:16px 10px; text-align:center; border-right:1px solid #eef2f7;"> <div style="font-size:26px; font-weight:800; color:#059669;">83</div> <div style="font-size:10.5px; color:#64748b; letter-spacing:0.8px;">TOOL-USE</div> </div> <div style="flex:1; min-width:110px; padding:16px 10px; text-align:center; border-right:1px solid #eef2f7;"> <div style="font-size:26px; font-weight:800; color:#7c3aed;">+8</div> <div style="font-size:10.5px; color:#64748b; letter-spacing:0.8px;">PP vs NARROW</div> </div> <div style="flex:1; min-width:110px; padding:16px 10px; text-align:center; border-right:1px solid #eef2f7;"> <div style="font-size:26px; font-weight:800; color:#db2777;">83</div> <div style="font-size:10.5px; color:#64748b; letter-spacing:0.8px;">TOK/S · MTP3</div> </div> <div style="flex:1; min-width:110px; padding:16px 10px; text-align:center;"> <div style="font-size:26px; font-weight:800; color:#334155;">26<span style="font-size:14px;">GB</span></div> <div style="font-size:10.5px; color:#64748b; letter-spacing:0.8px;">ON DISK</div> </div> </div> <div style="text-align:center; font-size:11px; color:#94a3b8; margin:6px 0 24px;">📊 All benchmarks &amp; throughput measured on a single <b>RTX 4090 (48 GB)</b> · vLLM 0.19 · MTP n=3 · seed 42</div>

The clean `Qwen/Qwen3.6-27B` base (no distill, no abliteration — creator alignment intact), quantized by us with AutoRound INT4 using a wide recipe: the large linear-attention (SSM/GDN) projections are kept in BF16 instead of INT4. That single choice lifts hard-coding from 70 → 78 (+8 pp) over the standard narrow quant — the highest hard score in our fleet, at a third the size of the 80B it beats. MTP head preserved → speculative decoding works in vLLM out of the box.

Highlights

<div style="display:grid; grid-template-columns:repeat(auto-fit,minmax(160px,1fr)); gap:10px; margin-bottom:26px; font-family:-apple-system,Segoe UI,Roboto,sans-serif;"> <div style="border:1px solid #a5f3fc; border-radius:10px; padding:14px; background:#ffffff;"> <div style="font-size:14px; font-weight:800; color:#0891b2;">🏆 Fleet record — hard 78</div> <div style="font-size:11.5px; color:#64748b; margin-top:4px; line-height:1.5;">our smartest build; beats the 80B Coder-Next (62) at ⅓ the size</div> </div> <div style="border:1px solid #ddd6fe; border-radius:10px; padding:14px; background:#ffffff;"> <div style="font-size:14px; font-weight:800; color:#6d28d9;">🧠 Clean base, no distill</div> <div style="font-size:11.5px; color:#64748b; margin-top:4px; line-height:1.5;">raw Qwen3.6-27B intelligence, not a fine-tune</div> </div> <div style="border:1px solid #bae6fd; border-radius:10px; padding:14px; background:#ffffff;"> <div style="font-size:14px; font-weight:800; color:#0369a1;">🔬 The "wide" trick: +8 pp</div> <div style="font-size:11.5px; color:#64748b; margin-top:4px; line-height:1.5;">big SSM projections kept BF16 → long-range logic survives</div> </div> <div style="border:1px solid #bbf7d0; border-radius:10px; padding:14px; background:#ffffff;"> <div style="font-size:14px; font-weight:800; color:#059669;">🧬 MTP preserved</div> <div style="font-size:11.5px; color:#64748b; margin-top:4px; line-height:1.5;">15-tensor head kept → speculative decoding just works</div> </div> <div style="border:1px solid #fbcfe8; border-radius:10px; padding:14px; background:#ffffff;"> <div style="font-size:14px; font-weight:800; color:#be185d;">🖼️ Multimodal</div> <div style="font-size:11.5px; color:#64748b; margin-top:4px; line-height:1.5;">images &amp; video — vision tower full BF16</div> </div> <div style="border:1px solid #a5f3fc; border-radius:10px; padding:14px; background:#ffffff;"> <div style="font-size:14px; font-weight:800; color:#0891b2;">📏 256k context (native)</div> <div style="font-size:11.5px; color:#64748b; margin-top:4px; line-height:1.5;">supports 256k; full 256k fits on 48 GB at mem 0.80</div> </div> <div style="border:1px solid #fde68a; border-radius:10px; padding:14px; background:#ffffff;"> <div style="font-size:14px; font-weight:800; color:#b45309;">⚡ 83 tok/s — brisk for dense</div> <div style="font-size:11.5px; color:#64748b; margin-top:4px; line-height:1.5;">MTP n=3; not a slow model despite being dense</div> </div> <div style="border:1px solid #cbd5e1; border-radius:10px; padding:14px; background:#ffffff;"> <div style="font-size:14px; font-weight:800; color:#0f172a;">🧾 Proof in-repo</div> <div style="font-size:11.5px; color:#64748b; margin-top:4px; line-height:1.5;">the per-task benchmark JSON ships with it</div> </div> </div>

Benchmarks — the wide trick pays off

Same clean base Qwen/Qwen3.6-27B, one harness — only the quant recipe changes. Keeping the big SSM projections in BF16 lifts hard +8 pp over the standard narrow quant:

+8 pp hard for +7.5 GB over narrow — and it beats Qwen3-Coder-Next **80B** (hard 62 on our harness), three times its size.

[!NOTE] Numbers are % of tasks passed (40 hard coding + 30 tool tasks; raw pass/total in `benchmark-results/`). Conditions: temp 1.0 / topp 0.95 / topk 20 / min_p 0 · seed 42 · non-thinking · vLLM 0.19 · RTX 4090 (48 GB) · MTP n=3. The abliterated & wide rows share the exact 40 hard tasks the narrow quant scored 70 on → clean apples-to-apples. The Lorbus row is our own measurement of their public quant, not their claim. The 80B Coder-Next row is likewise our own

measurement of the UD-IQ4_XS GGUF (via LM Studio) on the same task set — a bigger, different model shown for scale. Reproduce it, don't trust it.

🔬 The "wide" trick — why it scores higher

Qwen3.6 is a hybrid model: most layers are linear-attention (GDN/Mamba), where long-range logic flows through the large projections in_proj_qkv, in_proj_z, out_proj. Standard "narrow" INT4 quants compress these to 4-bit too — and that quietly costs reasoning. We keep them in BF16:

Everything else (full-attention q/k/v/o, MLP gate/up/down) is INT4. The recipe was validated on Qwen3.5 first (+6 pp) and confirmed here on 3.6 (+8 pp).

⚡ Speed — MTP speculative decoding

Honest 768-token × 3-run measurement on a single RTX 4090 (48 GB), vLLM 0.19:

Use `num_speculative_tokens: 3` — long-context throughput is already on its plateau (~40 tok/s) while acceptance stays highest (52%), so the draft head misfires least. Spec-decode is lossless — n changes speed only, never output.

🖼️ Multimodal — sees images & video

The full vision-language model. The entire vision tower (333 tensors) is kept in full BF16 — quantization never touched it — so image/video understanding is identical to the pristine base. Serve with --limit-mm-per-prompt '{"image":4,"video":1}'. Published benchmarks cover text/coding/tool-use; vision quality is inherited from the base, not separately re-scored.

🚀 Quickstart — this is a big-card model

Weights are 26 GB, so this is a big-card model. A 48 GB card is its home; a 32 GB card runs it but with only a small context window; 24 GB can't load it at all.

[!NOTE] 32 GB is tight. The 26 GB weights leave only ~1.7 GB for KV even at mem-util 0.95, so the safe window is ~16k tokens (measured by emulating a 32 GB budget on a 48 GB card: concurrency ~1.5x at 16k; 24k dropped to 1.03x = the hang zone). mem 0.95 assumes a dedicated/headless card — lower it if a display is attached. Want long context on ≤ 32 GB? Use the 19 GB **abliterated sister** (fits 24 GB with 40k ctx, or 32 GB with much more).

vllm serve antonyMox/Qwen3.6-27B-AutoRound-INT4-wide-MTP \
  --quantization auto-round \
  --max-model-len 150000 \
  --gpu-memory-utilization 0.72 \
  --max-num-seqs 1 \
  --kv-cache-dtype fp8_e4m3 \
  --enable-prefix-caching \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --trust-remote-code

Long context is cheap: Qwen3.6 is a hybrid (Mamba/GDN + attention), KV costs ~37 MB/1k tokens at fp8. The model's native maximum is 256k (262,144) tokens — how much fits on 48 GB depends on mem-util:

Context vs mem-util (48 GB card, measured)

Raise --gpu-memory-utilization toward 0.80 for the full 256k; the 0.72 default keeps more VRAM free (this variant's 26 GB weights leave less room than the 19 GB abliterated sister).

[!WARNING] Never set `--max-model-len` at exact KV capacity. On vLLM 0.19 hybrid models, requests in the last ~3 % of the window pass validation then hang forever (no error, GPU idle) — the scheduler can't allocate the final KV block. Keep the window a couple of blocks below capacity: after launch, the log line Maximum concurrency for N tokens: Yx must be ≥ ~1.10x (ours is 1.15x at 150k), never 1.00x. Also, the GPU KV cache size: X tokens log line understates real capacity ~3× on hybrids — trust only the Maximum concurrency line.

🧬 The family

Clean quants of the same base — pick by hardware:

The 27B sisters: AutoRound INT4, MTP preserved, multimodal. All benchmarked against the community reference Lorbus narrow INT4 (hard 70) — see the table above.

🍳 Recipe (reproducible)

1. 1.Base — pristine Qwen/Qwen3.6-27B BF16.
2. 2.Quant — AutoRound W4A16, group 128, 200 iters, v2 calibration.
3. 3.Wide fp_layers (kept BF16) — mtp.*, visual.*, small SSM (in_proj_a/b, A_log, conv1d, dt_bias, linear_attn.norm) + the wide trick: `in_proj_qkv`, `in_proj_z`, `out_proj`.
4. 4.MTP — 15 mtp.* tensors preserved (BF16), n=3 recommended.
5. 5.Sampler — ships temp 1.0 / topp 0.95 / topk 20 (generation_config.json).

🔢 Why HuggingFace shows "~11.5B params"

Not a small model — it's how the widget reads a partially-packed INT4 quant. The INT4 weight tensors are packed 8× int4 into each int32 (HF counts fewer slots), while the large wide projections, vision, and MTP stay full BF16 (counted fully). The true model is Qwen3.6-27B, 26 GB on disk. Verified breakdown: language_model 9.35 B · other 1.27 B · vision 0.46 B (333 t, BF16) · mtp 0.42 B (15 t, BF16). (The wide sister shows more BF16 than a narrow quant precisely because it keeps those big projections full-precision — that's the whole point.)

📜 License & attribution

Apache-2.0, same as the base. Quantized from `Qwen/Qwen3.6-27B` with a wide AutoRound INT4 recipe. This is the clean base (creator safety alignment intact). Community build; benchmark numbers are from our own harness — JSON included, methodology open. Provided "as is", without warranty.