srmiles/Qwen3.8-9B-Distill-MTP-GGUF

Qwen3.8-9B-Distill — MTP draft head (GGUF)

Standalone multi-token-prediction draft heads for `empero-ai/Qwen3.8-9B-Distill`, converted from the official BF16 safetensors with llama.cpp's own convert_hf_to_gguf.py.

The upstream model carries an MTP head in its weights (mtp_num_hidden_layers: 1 in config.json), but the official GGUF repo publishes main-model quants only. With no head file there is nothing to pass to --model-draft, so the model runs unassisted and gives up the speculative-decoding path it was built with. These files restore it.

Files

Both carry 18 tensors — the blk.32.nextn.* MTP block plus token_embd, output and output_norm — for 2.28B parameters, architecture qwen35.

sha256  5a3ac58e36407a0661a2c53cd629d644ff844ddbb7820c1e168be39351ec19ee  mtp-Qwen3.8-9B-Distill-head-BF16.gguf
sha256  cdc47bb91e8e149c43b3ff86bddd522491f6ef990aeaf9ffecbf9df15db22f80  mtp-Qwen3.8-9B-Distill-head-Q8_0.gguf

Why Q8_0 is the recommended default here. Draft heads are usually published unquantized, on the reasoning that a few hundred MB of head is negligible beside a multi-GB target and quantizing risks acceptance loss. That reasoning does not transfer to this model. Of the head's 2.28B parameters, ~2.03B is the vocab embedding and output matrices (248,320 x 4,096, twice) — not the MTP block itself. At BF16 the head is 4.56 GB against a 5.38 GB Q4KM target, so it very nearly doubles resident size. Q8_0 halves that while quantizing exactly the tensors that tolerate it best, and measured 81.4% acceptance — squarely in the band a same-architecture third-party head reaches. BF16 is here for anyone who wants to verify that trade or requantize differently.

Benchmarks

Intel Arc Pro B60 24 GB (Battlemage), llama.cpp SYCL with GGML_SYCL_F16=ON, build b10566 (bb4caa754), isolated card, Q4_K_M target + Q8_0 head.

Decode and acceptance over 20 runs x 300 tokens at sampled settings (temp 0.6 / top-p 0.95 / top-k 20) — not greedy, which is variance-dominated at small N. Prefill from server-side prompt_ms, 3 samples per size.

For scale, a same-architecture model with a well-regarded third-party MTP head — Ornith 1.5-9B Q4_K_M + Q8_0 head — measured 65.15 tok/s median and 84.7% acceptance on the same build, in the same session, under the same harness.

What the head is worth

Same model, same build, same session, with and without the head:

The head costs prefill to buy decode. +30.5% decode is the headline, but it also gives up 14.6% of prefill throughput and 3.85 GB of memory. That is clearly the right trade for chat and agent workloads, which are decode-bound. It is not automatically right for prefill-heavy workloads with short outputs — classification, routing, extraction — where the prefill loss can outweigh the decode gain. Measure your own workload shape before assuming the drafter is a win.

Decode with the head is bimodal, clustering near 74 and near 50 tok/s. Unassisted it is not (sigma 0.05), so the variance is acceptance-driven — the head is very good on some content and less good on other content — rather than thermal or scheduler noise. Prefer the median over the mean when comparing.

Tuning --spec-draft-n-max

Benchmarked at the conventional 3. On a different model on this hardware — one whose head reaches 99.5-100% acceptance — sweeping this flag was worth +16.3%, with a sharp throughput cliff once the verification batch (n_max + 1) exceeded 8. This head's acceptance is 81.4%, so it has less headroom, but the flag is worth sweeping rather than assuming. Do not tune it by stepping upward until throughput stops improving: on this backend that procedure can walk straight past a peak into a much worse setting.

Usage

llama-server \
  -m Qwen3.8-9B-Q4_K_M.gguf \
  --model-draft mtp-Qwen3.8-9B-Distill-head-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  -ngl 99 -ngld 99 \
  -c 262144 --parallel 1 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  -fa on -ub 2048 -b 2048 \
  --jinja --reasoning off \
  --top-k 20 --min-p 0.0 \
  --host 0.0.0.0 --port 8000

• —--spec-type draft-mtp — the MTP-native speculative path, not classic n-gram drafting.
• —-ngld 99 — offload the draft head too; leaving it on CPU negates the win.
• —--reasoning off — routes thinking tokens back into content. Mandatory for structured-JSON workloads, which otherwise get an empty content and the text in reasoning_content.
• —--spec-draft-n-max 3 — matches the model's single MTP layer. Raising it past what the head was trained for does not help.

Check acceptance on your own workload from the server's Prometheus endpoint — llamacpp:spec_decode_num_accepted_tokens_total over llamacpp:spec_decode_num_draft_tokens_total. Acceptance is workload-dependent; a number measured on synthetic filler is an upper bound, not a promise.

Reproducing the conversion

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
python -m venv .venv && .venv/bin/pip install \
  --index-url https://download.pytorch.org/whl/cpu torch
.venv/bin/pip install numpy sentencepiece transformers protobuf safetensors
.venv/bin/pip install -e gguf-py

hf download empero-ai/Qwen3.8-9B-Distill --local-dir Qwen3.8-9B-Distill-hf

.venv/bin/python convert_hf_to_gguf.py Qwen3.8-9B-Distill-hf \
  --mtp --outtype bf16 --outfile mtp-Qwen3.8-9B-Distill-head-BF16.gguf

.venv/bin/python convert_hf_to_gguf.py Qwen3.8-9B-Distill-hf \
  --mtp --outtype q8_0 --outfile mtp-Qwen3.8-9B-Distill-head-Q8_0.gguf

--mtp exports only the MTP tensors as a standalone draft GGUF. Its counterpart --no-mtp writes the target without them, so a repo can publish both halves cleanly. The same script also has --dspark for DSpark-style heads.

Converted at commit bb4caa754. Any recent llama.cpp with qwen35 MTP support will load these.

License

Apache 2.0, inherited from `empero-ai/Qwen3.8-9B-Distill`, itself a distillation of `Qwen/Qwen3.5-9B`. No weights were modified — this is a format conversion and quantization of tensors already present in the upstream release.

Not affiliated with empero-ai or Qwen. Thanks to both for publishing open weights, and to the llama.cpp maintainers for the MTP export path.