Modelpublic

localweights/Qwen3.6-35B-A3B-MTP-IQ4_XS-Q8nextn-GGUF

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face

6likes215downloads

Model Card

Qwen3.6-35B-A3B-MTP-IQ4_XS-Q8nextn-GGUF

Qwen3.6-35B-A3B (MoE, 3B active) with NextN/MTP head preserved. Body quantized to IQ4_XS, NextN head kept at Q8_0 — leaner trunk + higher-precision draft head. Built via the patched convert_hf_to_gguf.py from patched llama.cpp build with Qwen3.5/3.6 MTP support.

Files

File	Size	Purpose
`Qwen3.6-35B-A3B-MTP-IQ4_XS-Q8nextn.gguf`	19.4 GB	Body IQ4XS + NextN head Q80

Why Q8nextn

Pure IQ4XS quantizes the NextN head along with the body. Keeping the head at Q80 preserves draft prediction quality and lets you push --spec-draft-n-max deeper without accept-rate collapse.

bash

llama-quantize \
  --tensor-type "blk\..*\.nextn\..*=q8_0" \
  Qwen3.6-35B-A3B-MTP-bf16.gguf \
  Qwen3.6-35B-A3B-MTP-IQ4_XS-Q8nextn.gguf \
  IQ4_XS

Marginal size cost (~+0.4 GB), measurable speed + accept gain at deeper drafts.

Optimal serving config (RTX 3090 Ti)

bash

llama-server -m Qwen3.6-35B-A3B-MTP-IQ4_XS-Q8nextn.gguf \
  -ngl 999 -fa on \
  --spec-type mtp --spec-draft-n-max 6 \
  --no-mmap \
  --ctx-size 200000 -ctk q4_0 -ctv q4_0 \
  --parallel 1 --kv-unified \
  --metrics --jinja

Performance — `--spec-draft-n-max` sweep

Measured 2026-05-06 on Crucible (9950X, 96 GB DDR5-4800 dual-channel, RTX 3090 Ti). Prompt: count 1→50, 300-token decode.

IQ4_XS-Q8nextn (this repo)

n	Decode tok/s	Accept rate
1	205	100%
2	241	100%
3	256	99%
4	277	99%
5	276	93%
6	299 ← peak	98%
7	235	70%

Sibling IQ4_XS pure (for comparison)

n	Decode tok/s	Accept rate
4	275	97%
5	282 ← peak	96%
6	276	90%

Q8nextn beats pure peak by ~6% AND tolerates n=6 with higher accept (98% vs 90%).

Context capacity (RTX 3090 Ti, 24 GB)

ctx	VRAM
4 K	20.6 GB
64 K	21.0
192 K	22.0
256 K (max)	22.8

KV cost negligible due to GQA-4 + Q40 KV + `kvonly_nextn`.

Sibling repos

License

Apache 2.0.

Provenance

Built on Crucible: 9950X / 96 GB DDR5 / RTX 3090 Ti.