leonsarmiento/Qwen3.8-27B-3bit-mtp-mlx
leonsarmiento/Qwen3.8-27B-3bit-mtp-mlx
Qwen3.8-27B VLM with the MTP drafter baked in — the speed variant of this quantization family. Identical 3-bit mixed recipe as Qwen3.8-27B-3bit-mlx, plus the source model's Multi-Token Prediction layer bundled inside the checkpoint, so the engine runs speculative decoding natively — no separate drafter model to download or pair.
Which one should you get?
⚠️ Lightning MTP speedup is an oMLX-only feature (verified on oMLX v0.6.1, build 2323). LM Studio does not execute bundled MTP layers — this model loads and runs fine there, but the MTP tensors are inert dead weight. LM Studio users: get the clean [Qwen3.8-27B-3bit-mlx](https://huggingface.co/leonsarmiento/Qwen3.8-27B-3bit-mlx) instead. MTP is also not needed for thinking mode — see below.
Quantization Details
Same recipe as the clean build:
- Embedding layers: 4-bit with group_size=64
- Prediction head (lm_head): 4-bit with group_size=64
- Vision tower: 8-bit with group_size=64
- All other layers: 3-bit with group_size=64
MTP add-on (extracted from the source mtp.* tensors, scottlowry/oQ checkpoint convention):
- 7 norm tensors: bf16, +1.0 MLX RMSNorm convention
fc.weight: bf16 unquantized (bit-identical to source)- 7 linear layers: 4-bit affine, group_size=64 (per-tensor config overrides — the 3-bit default doesn't apply to them)
text_config.mtp_num_hidden_layers: 1,mtp_use_dedicated_embeddings: false
Stats: 3.716 bpw base + 0.33 GB MTP · 13.04 GB · 3 shards · mlx-vlm · vision tower preserved · chat template with reasoning_effort defaulting to low.
Benchmarks (oMLX v0.6.1, Engine Auto, Context: Code/Python, pp1024/tg128)
- +30.7% single-stream decode with MTP on (15.3 → 20.0 tok/s)
- Beats the 4-bit oQ4e reference at batch ≥ 2 (90.9 vs 87.2 tok/s @ 4×) while using 2.66 GB less memory
- Single-stream sits ~10% below a 4-bit verifier: the 3-bit base rejects more speculative drafts (verifier noise) — the accepted trade for the smaller footprint
- Disable TurboQuant KV cache — MTP shows no advantage with it enabled
When to turn MTP off: thinking/reasoning-heavy tasks. Faster token generation gets spent on more reasoning, not faster answers (measured on Qwen3.6-35B MATHQA: −7pp accuracy with MTP on, +20% wall time). With MTP off this model behaves exactly like the clean 3-bit build.
Source
Qwen/Qwen3.8-27B — dense hybrid-attention VLM, 64 layers (48 linear attention + 16 full attention), 262K context, MTP prediction layer.
Instruct Benchmarks
Instruct mode (thinking off). Sampled: MMLU 100/14042, MMLU_PRO 100/12032, HUMANEVAL 50/164, MBPP 50/500. Wall times not directly comparable across architectures (dense vs MoE active params).
Highlights: HUMANEVAL 94% at dense 3-bit — strong code for the size class. Benchmarks ran in instruct mode; engage Lightning MTP (oMLX only) for decode speed, not for these quality numbers.