localweights/Qwen3.6-35B-A3B-MTP-IQ4_XS-GGUF

Qwen3.6-35B-A3B-MTP-IQ4_XS-GGUF

Qwen3.6-35B-A3B (MoE, 3B active parameters) with NextN/MTP (Multi-Token Prediction) speculative-decoding head, quantized to IQ4_XS for single-GPU 24 GB inference.

What this is

A self-contained GGUF carrying both the trunk and the NextN MTP head. With a patched llama.cpp it serves at ~2× decode of the same trunk without MTP, and at long contexts the MoE+MTP combination wins on every dimension we measured: faster than the dense 27B, better long-ctx behavior, lower VRAM footprint than non-MTP equivalents.

This is what we use as our "orchestrator" / chair model for council pipelines on a 24 GB card.

Build pipeline

Source: Qwen/Qwen3.6-35B-A3B (HF safetensors, with NextN tensors).

1. 1.Clone llama.cpp at the crucible-mtp branch on llama.cpp (patched) (Aman Gupta's MTP fork) — adds LLM_ARCH_QWEN35MOE_MTP + the NextN draft path.
2. 2.Run convert_hf_to_gguf.py against the HF repo. Produces a BF16 GGUF with arch qwen35moe_mtp and the NextN tensors fused in.
3. 3.Quantize to IQ4_XS via llama-quantize.

python convert_hf_to_gguf.py /path/to/Qwen3.6-35B-A3B \
    --outfile Qwen3.6-35B-A3B-MTP-bf16.gguf
llama-quantize Qwen3.6-35B-A3B-MTP-bf16.gguf \
    Qwen3.6-35B-A3B-MTP-IQ4_XS.gguf IQ4_XS

Optimal serving config (RTX 3090 Ti, 24 GB)

Cherry-pick PRs #20819 + #20822 for cross-process KV-slot save/restore (sub-second resume on long contexts — material when this model is used as a long-running chair / orchestrator).

llama-server \
  -m Qwen3.6-35B-A3B-MTP-IQ4_XS.gguf \
  -ngl 999 -fa on \
  --spec-type mtp --spec-draft-n-max 4 \
  --no-mmap \
  --ctx-size 200000 \
  --batch-size 1024 --ubatch-size 512 \
  -ctk q4_0 -ctv q4_0 \
  --parallel 1 --kv-unified \
  --ctx-checkpoints 8 --checkpoint-every-n-tokens 2048 \
  --cache-ram -1 --cache-idle-slots \
  --metrics --jinja

Why these flags:

• —--spec-type mtp: enables NextN-head draft path. The whole point of the MTP variant.
• —--spec-draft-n-max 4: empirical sweet spot. n=2 underutilises the head; n=6+ drops accept rate faster than throughput grows. n=4 lands at ~85% accept × ~101 tok/s.
• —--ctx-size 200000: sized to leave ~1.5 GB VRAM headroom on 24 GB at q4/q4 KV. Native max is 262 K — push to that only if you don't need headroom for activations.
• —--no-mmap: required for KV-slot persistence and zero perf hit on this rig.
• —-ctk q4_0 -ctv q4_0: dense KV cache; q8 KV at this ctx exceeds 24 GB during warmup.
• —--parallel 1: MTP path is single-stream upstream.

What NOT to set:

• —-ot "exps=CPU" (expert offload) — kills speedup; only consider if running alongside another model.
• —-ctk q8_0 at full 200K ctx — VRAM will OOM at warmup.

Performance (RTX 3090 Ti, 350 W power limit)

Measured 2026-05-06 with persistence + MTP on:

Memory footprint at 200 K ctx: ~18.5 GB (model) + ~3 GB (KV q4/q4) + scratch ~22.5 GB. Headroom for activation spikes.

Tokenizer

Qwen3.6 tokenizer (248,320 vocab, qwen35 pre-tokenizer). Same chat template as upstream Qwen3.6-35B-A3B-Instruct. If your runtime errors on Jinja Exception: System message must be at the beginning, use a loosened jinja template (single line: replace the strict-position raise with a no-op).

Why MTP matters here

Qwen3.6-35B-A3B in plain GGUF form decodes at ~50 tok/s on this rig (single-stream MoE, 3B active). With MTP enabled and ~85% accept rate at draft n=4, we observe ~100 tok/s — exactly the 2× the NextN paper claims. On the 3090 Ti this is the difference between "useful chair model" and "infuriatingly slow."

The MoE-MTP combination also has a non-obvious benefit: long-context decode degrades much more gracefully than the 27B dense at the same VRAM budget. At 190 K tokens, 35B-A3B-MTP holds ~50 tok/s where 27B drops below 30.

License

Apache 2.0 (inherited from Qwen3.6).

Provenance

Built on Crucible: 9950X / 96 GB DDR5 / RTX 3090 Ti. Sibling: localweights/Qwen3.6-27B-MTP-IQ4_XS-GGUF.