froggeric/Qwen3.6-27B-MTP-GGUF

<div align="center">

Qwen3.6-27B with MTP

Up to 2.7× faster with MTP · 262K context on 48 GB · Fixed chat template

Dense 27B model with vision, thinking, and tool use — self-speculative decoding, \ configurable KV cache, fixed Jinja template (tool calls and thinking actually work in C++ runtimes), \ and a server with both OpenAI and Anthropic APIs.

</div>

Start the server

You need llama.cpp b9180 or newer (released 2026-05-16, includes MTP support). Install via Homebrew:

brew install llama.cpp

llama-server -m Qwen3.6-27B-Q5_K_M-mtp.gguf \
  --mmproj mmproj-Qwen3.6-27B-f16.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  -c 262144 -fa off --n-predict -1 --temp 0.6 --top-p 0.95 --top-k 20 --repeat-penalty 1.0 \
  -ngl 99 --port 8081

Sampling is set for coding tasks (temp 0.6, top_p 0.95). Adjust -m and -c for your hardware — see the quant table below. For general chat, change to --temp 0.7 --top-p 0.80. Drop --mmproj if you don't need vision.

Optional flags

8-bit KV cache — halves KV memory at minor quality cost. Use when f16 KV doesn't give enough context:

  --cache-type-k q8_0 --cache-type-v q8_0

Custom chat template — override the embedded template. Use this if your runtime doesn't support the bundled Jinja template, or if you need the official Qwen template instead of the fixed one:

  --jinja --chat-template-file chat_template.jinja

MTP Speculative Decoding — Should You Enable It?

MTP predicts extra tokens per step using the model's own MTP heads, then verifies them in one pass. No extra model or VRAM needed — it's built into the weights. But it doesn't help equally for everything.

What controls the speedup is not your quant or temperature — it's what you're generating.

Recommendation Matrix

🟢 speeds up · 🟡 marginal · 🔴 slower with MTP

Rules of thumb:

• —Q8_0 and F16: always enable MTP — even creative writing gets +48–67%
• —Coding at any quant: keep it on
• —Q4_K_M–Q6_K creative tasks: turn it off (--spec-type none)

Why Task Type Dominates

Draft token acceptance by task type (percentage of predicted tokens that are correct):

A 40-point spread from code to creative. Temperature (0.0–0.7) and quant level barely move the needle. What you're generating matters 40× more than any other setting.

Speedup by Quant × Task

Measured on M2 Max 96 GB, temp 0.7, N=3 draft tokens, long generation (2500 tokens):

Why does F16 benefit most? F16 at 51 GB crawls at 6.6 tok/s because every token means dragging the full model through memory. Accepted MTP drafts skip that expensive pass. Q4KM at 16 GB is already fast enough that the draft overhead is barely worth it on anything less predictable than code.

Draft Token Count

N=3 is optimal for all quants except F16 (where N=4 edges ahead: 17.9 vs 16.2 tok/s). Higher values waste compute on rejected tokens. Lower is too conservative.

Thinking Mode

With thinking enabled for coding tasks, Q8_0 draft acceptance drops from 87% to 73%. Still +94% speedup — keep MTP on.

<details> <summary>About these numbers</summary>

The comprehensive table above was measured with the original MTP implementation (llama.cpp PR #22673, the custom build that first added MTP support). Current mainline llama.cpp (b9180+, including Homebrew) gives ~10–17% lower MTP speedup due to implementation differences. Verified on mainline b9260:

The recommendation matrix above is based on relative patterns that are identical on both builds — the advice doesn't change.

</details>

Which quant should I download?

Find your hardware below — each row gives the best quant, KV cache type, and max context that fits.

Apple Silicon

Qwen3.6-27B is a hybrid model — only 16 of 65 layers use KV cache (verified). The other 48 are linear attention (fixed 150 MiB recurrent state). KV memory is ~4× less than a standard dense model. Runtimes that don't handle this (e.g. vllm) allocate KV for all 65 layers and show much higher memory usage.

Numbers below include all measured overhead (GPU compute buffers, CPU model/compute buffers — ~4% of total). Must leave ≥ 8 GB for macOS (24 GB Macs: 6 GB; 16 GB Macs: 4 GB). Plus 2 GB safety margin.

NVIDIA GPU

Same model memory as Apple Silicon, plus ~1 GB CUDA overhead. Numbers include 2 GB safety margin.

Quick picks: 16 GB Mac → IQ2_M · 24 GB Mac → IQ3_M · 32 GB Mac → Q4_K_M · 36 GB Mac → Q5_K_M · 48 GB Mac → Q6_K · 64 GB Mac → Q8_0 · 96 GB+ Mac → F16 Leave KV cache at f16 (blank column) for best quality. Use q8_0 KV only when f16 doesn't give enough context. q4_0 KV should not exceed 64K context. Vision adds ~0.9 GB for mmproj. macOS needs ≥ 8 GB for itself (24 GB Macs: 6 GB; 16 GB Macs: 4 GB). You can increase available memory: sudo sysctl iogpu.wired_limit_mb=90112 (88 GB on a 96 GB Mac). NVIDIA reserves ~1 GB for CUDA.

API usage

The server provides both OpenAI and Anthropic APIs.

OpenAI-compatible (/v1/chat/completions)

curl http://localhost:8081/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen","messages":[{"role":"user","content":"Hello"}]}'

Works with any OpenAI client — point it at http://localhost:8081/v1.

Anthropic-compatible (/v1/messages)

curl http://localhost:8081/v1/messages \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen","max_tokens":1024,"messages":[{"role":"user","content":"Hello"}]}'

Works with any Anthropic client. Supports streaming, tool use, and vision.

Claude Code

ANTHROPIC_BASE_URL=http://127.0.0.1:8081 claude

Tool use

curl http://localhost:8081/v1/messages \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen",
    "max_tokens": 1024,
    "tools": [{
      "name": "get_weather",
      "description": "Get current weather for a location",
      "input_schema": {
        "type": "object",
        "properties": {"location": {"type": "string"}},
        "required": ["location"]
      }
    }],
    "messages": [{"role": "user", "content": "What is the weather in Paris?"}]
  }'

Vision

The main server command above already includes --mmproj. Just send an image:

curl http://localhost:8081/v1/messages \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": [
      {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": "'$(base64 < photo.jpg)'"}},
      {"type": "text", "text": "Describe this image"}
    ]}]
  }'

Note: Vision + MTP works on llama.cpp b9240+. Older builds (PR #22673) crashed when combining vision with MTP — fixed in mainline.

Direct CLI usage

llama-cli -m Qwen3.6-27B-Q5_K_M-mtp.gguf \
  --mmproj mmproj-Qwen3.6-27B-f16.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  -c 4096 -n 2048 --temp 0.6 --top-p 0.95 --top-k 20 --repeat-penalty 1.0 \
  -ngl 99 -p "Your prompt here"

Downloads

All tiers include MTP heads and were quantized directly from the F16 conversion for maximum precision. I-quant tiers (IQ4XS, IQ3M, IQ2M) use unsloth's importance matrix. **Q5KM** is the sweet spot — use Q4KM if you're tight on RAM, Q80 for high quality, or F16 for long agentic coding sessions where quantization artifacts compound noticeably. GPU means NVIDIA (RTX 3060 = 12 GB, RTX 3090/4090 = 24 GB, A6000 = 48 GB, A100 = 80 GB).

Hardware numbers assume f16 KV for "Min." (4K) and q8_0 KV for "Recommended" (80K) and "Max" (262K).

System prompt & sampling

System prompt

The first line must be:

You are Qwen, created by Alibaba Cloud. You are a helpful assistant.

The model underperforms without it. Append anything after that line.

Thinking toggle

Drop <|think_on|> or <|think_off|> in any message to toggle thinking. The template strips the tag so the model never sees it.

Sampling

From the official Qwen authors. Reserve 128K+ context for thinking mode.

Compatibility

LM Studio users: use the MLX 8-bit or MLX 4-bit instead — full vision + tools + thinking, no MTP.

Other speculative decoding modes

Draft model (separate small model)

Pair with a smaller Qwen 3.5/3.6 model that shares the same tokenizer. Can give ~2.3× speedup.

llama-cli -m Qwen3.6-27B-Q5_K_M-mtp.gguf \
  -md Qwen3.5-0.8B-Q8_0.gguf \
  --spec-draft-n-max 10 -ngl 99 -ngld 99 \
  -c 4096 -n 2048 --temp 0.7 \
  -p "Your prompt"

ngram-mod (no extra model, benefits repeat prompts)

Uses cached n-grams from previous prompts. Works best for repeated/similar prompts.

--spec-type ngram-mod \
--spec-ngram-mod-n-match 24 \
--spec-ngram-mod-n-min 48 \
--spec-ngram-mod-n-max 64 \
--repeat-penalty 1.0

<details> <summary>Memory requirements (detailed)</summary>

Approximate VRAM on Apple Silicon (unified memory), using Q5KM as reference. Includes 150 MiB recurrent state (constant, does not scale with context) plus ~1.5 GB compute/CPU overhead. Only 16 of 65 layers use KV cache — the other 48 use linear attention. Numbers are measured from actual allocations, not estimates.

"Total" = model + KV cache + recurrent state + compute/CPU overhead. macOS needs ≥ 8 GB (24 GB Macs: 6 GB; 16 GB Macs: 4 GB). With vision: add 0.9 GB for the mmproj.

KV cache options

Recommendation: Leave KV at f16 for best quality. Use q8_0 when f16 doesn't give enough context. Reserve q4_0 for tight RAM — and only up to 64K context.

Memory per quant tier (4K context, f16 KV)

</details>

<details> <summary>Chat template fixes</summary>

The bundled Jinja template fixes several bugs in the official Qwen 3.6 template:

• —Tool calls crash on C++ engines. The official template uses Python's |items filter and |safe, which don't exist in C++ Jinja runtimes (llama.cpp, LM Studio). This template uses direct dictionary key lookups.
• —The `developer` role crashes. Modern APIs send message.role == "developer". The official template throws an exception. This template maps it to system.
• —Empty `preserve_thinking` spam. The official template wraps every past turn in empty <think/> blocks, wasting context tokens. This template only emits thinking blocks with actual content.
• —`</thinking>` hallucination handling. The model sometimes generates </thinking> instead of the expected closing tag. Both are handled gracefully.

See Qwen-Fixed-Chat-Templates for the standalone template repo.

Note: The fixed template works in llama.cpp but may cause errors in some frameworks (oh-my-pi, Codex, etc.) — typically Jinja Exception: System message must be at the beginning. If you hit this, use the default (unfixed) template instead.

</details>

<details> <summary>Architecture details</summary>

</details>

<details> <summary>Conversion details</summary>

Converted from official Qwen3.6-27B safetensors using mainline convert_hf_to_gguf.py from llama.cpp (b9180+, Homebrew v9240). MTP tensors are included by default — no custom build needed. The fixed chat template (v19) from Qwen-Fixed-Chat-Templates was embedded in tokenizer_config.json before conversion.

Quantization source: F16 (not Q80) — all tiers are quantized directly from the F16 conversion for maximum precision, avoiding double-quantization artifacts. Standard K-quant tiers (Q80, Q6K, Q5KM, Q4KM) use no importance matrix. I-quant tiers (IQ4XS, IQ3M, IQ2M) use unsloth's importance matrix (calibrated with chat template at 6K–12K context, 76 chunks, 496 entries). IQ2M keeps MTP tensors at Q4K since the importance matrix doesn't cover MTP layer tensors.

# Prerequisites
brew install llama.cpp
git clone --depth 1 --filter=blob:none --sparse https://github.com/ggml-org/llama.cpp.git llama.cpp-source
cd llama.cpp-source && git sparse-checkout set convert_hf_to_gguf.py conversion gguf-py
python3 -m venv .venv && .venv/bin/pip install torch numpy tqdm transformers sentencepiece pyyaml requests
.venv/bin/pip install -e gguf-py

# Embed fixed chat template (v19) into source tokenizer_config.json
python3 -c "
import json
with open('Qwen/Qwen3.6-27B/tokenizer_config.json') as f: d = json.load(f)
with open('Qwen-Fixed-Chat-Templates/chat_template_oneline.txt') as f: t = f.read().strip()
d['chat_template'] = t
with open('Qwen/Qwen3.6-27B/tokenizer_config.json', 'w') as f: json.dump(d, f, indent=2, ensure_ascii=False)
"

# Convert to F16 (text + MTP, ~51 GB, ~30-40 min)
.venv/bin/python convert_hf_to_gguf.py Qwen/Qwen3.6-27B/ \
  --outtype f16 --outfile Qwen3.6-27B-F16-mtp.gguf --verbose

# Extract vision encoder
.venv/bin/python convert_hf_to_gguf.py Qwen/Qwen3.6-27B/ \
  --outtype f16 --mmproj --outfile mmproj-Qwen3.6-27B-f16.gguf --verbose

# Quantize K-quant tiers from F16 (no imatrix)
F16=Qwen3.6-27B-F16-mtp.gguf
llama-quantize $F16 Qwen3.6-27B-Q8_0-mtp.gguf  Q8_0
llama-quantize $F16 Qwen3.6-27B-Q6_K-mtp.gguf  Q6_K
llama-quantize $F16 Qwen3.6-27B-Q5_K_M-mtp.gguf Q5_K_M
llama-quantize $F16 Qwen3.6-27B-Q4_K_M-mtp.gguf Q4_K_M

# Quantize I-quant tiers from F16 with unsloth imatrix
IMATRIX=imatrix_unsloth.gguf_file
llama-quantize --imatrix $IMATRIX $F16 Qwen3.6-27B-IQ4_XS-mtp.gguf IQ4_XS
llama-quantize --imatrix $IMATRIX $F16 Qwen3.6-27B-IQ3_M-mtp.gguf  IQ3_M
# IQ2_M: MTP tensors at Q4_K (imatrix doesn't cover them)
llama-quantize --imatrix $IMATRIX --tensor-type blk.64.=q4_K $F16 Qwen3.6-27B-IQ2_M-mtp.gguf IQ2_M

</details>

Links

• —Original model
• —MLX 8-bit (LM Studio, Apple Silicon native, no MTP)
• —MLX 4-bit
• —Fixed chat templates
• —Qwen3.6 blog post
• —MTP benchmark details

Authorship

License

Apache-2.0, inherited from Qwen3.6.