fraQtl/Qwen3.6-35B-A3B-Hi-Fi-MTP-runtime

Qwen 3.6 35B-A3B Hi-Fi — MTP runtime variant

Runtime-verified. Same calibration as the Hi-Fi Q4KM variant, with the MTP draft head (blk.40) retained for native speculative decoding in llama-server.

🖥️ Datacenter target (A100-80GB). The speedup below is a datacenter-GPU result. On Apple Silicon / consumer GPUs there is no measured speedup — for local runs use the Hi-Fi GGUF.

One-line claim

Qwen 3.6 35B-A3B MTP runtime path verified in llama-server: ~76.0% draft acceptance (95% CI 75.7–76.3%) and ~1.49× decode speedup (95% CI 1.46–1.52×) on a 50-prompt chat-template suite, A100-80GB, k=4. 3-seed lock (seeds 42, 1337, 2024).

This is runtime verification, not a calibrated draft-head claim. The MTP block inherits the Hi-Fi main-model quant policy plus one Q80 override on `nextn.ehproj`.

Requirements

• —llama-server with Qwen 3.5/3.6 NextN MTP loader (llama.cpp main commit 2f6c815dc… or later)
• —Send requests via the /v1/chat/completions endpoint (NOT /completion — see caveat below)

How to run

llama-server \
  -m qwen36-35b-a3b-hi-fi-mtp-runtime.gguf \
  -ngl 999 -c 4096 \
  --spec-type draft-mtp \
  --spec-draft-n-max 4 --spec-draft-n-min 1

Request body:

{
  "messages": [{"role": "user", "content": "Write a Python function ..."}],
  "max_tokens": 256,
  "temperature": 0,
  "speculative": {"type": "draft-mtp", "n_max": 4, "n_min": 1}
}

Inspect acceptance via timings.draft_n (proposed) and timings.draft_n_accepted (accepted).

Measured numbers (3-seed lock, 2026-05-29)

Vanilla (non-spec) decode baseline: 133.61 tok/s. Hardware: A100-80GB. Context: 4K. npredict: 64. Greedy (temp=0, topk=1).

Per-seed: seed 42 → 75.97%/1.472× · seed 1337 → 76.14%/1.502× · seed 2024 → 75.85%/1.493×.

Draft-window sensitivity

k=4 is the measured optimum. Higher k collapses per-position acceptance because deeper drafts compound on the head's own (potentially wrong) previous predictions. Lower k underuses the draft window.

Caveats

• —Invocation-sensitive. Sending raw prompts to /completion (no chat template) drops acceptance to ~63% and speedup to 1.30×. Use /v1/chat/completions or apply the Qwen chat template manually.
• —Datacenter artifact — A100-80GB only. The 1.49× speedup is measured on A100-80GB at 4K context. On consumer hardware (Apple Silicon / M-series, single consumer GPUs) the A3B MoE is compute-bound, so speculative verification yields no measured net speedup — MTP on those targets is a compatibility smoke, not a speed win. For local / Apple Silicon use, run the [Hi-Fi GGUF](https://huggingface.co/fraQtl/Qwen3.6-35B-A3B-Hi-Fi-GGUF) instead (same calibration, no MTP head, no special runtime). Longer contexts may also shift the speedup.
• —Requires `llama-server`. llama-cpp-python (as of 2026-05-29) does not expose the MTP draft path. Bare llama-cli does not run server-side speculative decoding.
• —Runtime-verified, not calibrated. The MTP block uses the main-model imatrix (which has no activation statistics for blk.40). Whether a calibrated draft head improves acceptance is open research.

Provenance

• —Base model: unsloth/Qwen3-Next-80B-A3B-Instruct-GGUF BF16 revision 5c2410d71524f4f72b023ce8daf7a80528226d5f (MTP-inclusive)
• —Imatrix: imatrix_codemath_v1_256k.dat (sha 5872a78f610050d2fccdce0c13ae450a472647c9fb297fe0a7ccaf2dfa945460) — calibrated on the code+math corpus, 256K tokens
• —llama.cpp: main commit 2f6c815dc450106ef877ae32f4472bfd5cf83e47
• —Artifact sha256: 457114c45cd1b918ce71cfe01dcc7f4f70a0853092c911d9c6760ef16e25443c

All measurements + sensitivity probes are reproducible from the receipts under receipts/.

Related

• —Hi-Fi (main model): `fraQtl/Qwen3.6-35B-A3B-Hi-Fi-GGUF` — same calibration, no MTP head.
• —D1 (Mistral 7B fraQtl sidecars): `fraQtl/Mistral-7B-v0.3-fraqtl-sidecars`.

License

Inherits Qwen 3.6 license from the upstream base model.

More from fraQtl

The serving lane — KV-cache compression sidecars for vLLM — holds nine concurrent ≈128K-context users on a single A100 (134.1 tok/s aggregate, 9/9 per-user retrieval checks, receipt 2026-08-14): fraQtl/qwen3-4b-instruct-2507-kv-sidecars. Org page: huggingface.co/fraQtl.