dzannotti/Qwen3.8-Flash-Next-MTP-GGUF

Qwen3.8-Flash-Next MTP draft head (GGUF, standard quant types)

The multi-token-prediction head that ships inside Qwen/Qwen3.8-Flash-Next: one full qwen4exp block (attention + 512-expert MoE + hyper-connections, 4B parameters), trained jointly with the model. The upstream converter (PR 27742) drops it, so the popular GGUFs have none. This one is exported straight from the bf16 checkpoint and quantized with plain llama-quantize, so it runs on stock llama.cpp kernels (any backend), not a fork's quant types.

This is a draft head, not a model. Use it as -md next to any Flash-Next GGUF target (tested with unsloth's UD-Q4KXL and UD-IQ4_XS).

Needs a patched llama.cpp (for now)

qwen4exp is not upstream yet and neither is its MTP graph. Two options:

• —llama.cpp `7584430716ee` (tag b10612) + PR 27742 (model: add Qwen3.8-Flash-Next, pinned at unslothai/llama.cpp@bea3b12d) + `patches/qwen4exp-mtp-draft-head.patch` from this repo. The patch is the draft-head graph (JJJYmmm's PR 27739, reconciled by LaurentZuijdwijk) and the converter's --mtp export, plus one fix (an MTP-only export emits no PLE keys). Verified to apply and build on exactly that tree.
• —Or build LaurentZuijdwijk/llama.cpp branch vulkan/qwen4exp-rocmfpx, which carries the same two commits (b98aa9847, ae96a0dc0) on a Vulkan backend tuned for Strix Halo.

Set LLAMA_ATTN_ROT_DISABLE=1: upstream's quantized-KV activation rotation (#21038) is not supported by the qwen4exp attention path and the server aborts at load without it.

Run

LLAMAATTNROTDISABLE=1 llama-server \ -m Qwen3.8-Flash-Next-UD-Q4KXL-00001-of-00004.gguf \ -md Qwen3.8-Flash-Next-MTP-Q4KM.gguf -ngld 999 \ --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.75 \ -ngl 999 -fa on -ctk q80 -ctv q8_0 -c 262144

p-min 0.75 stops drafting when the head is unsure; that matters more than depth. Deeper fixed drafting (n-max 8, p-min 0.3) measured slower, and so did adaptive drafting: this head carries its own MoE, so every drafted token is a real forward pass.

The target's 28.8 GB n-gram table (per_layer_token_embd) is the model's problem, not the head's: upstream keeps it in host memory (-ot per_layer_token_embd=CPU), Laurent's fork splits it per head so it fits a Vulkan device. Our numbers below were taken with a patch that leaves it on disk and reads rows per batch; it changes memory, not speed.

Attached to unsloth's UD-Q4KXL (no -md needed)

unsloth-UD-Q4_K_XL-mtp-shards/ turns unsloth/Qwen3.8-Flash-Next-GGUF's UD-Q4KXL into a model that carries its own draft head, the way Qwen3.5/3.6-MoE GGUFs do (nextn_predict_layers = 1, the head as blk.48.*). Two files, 1.9 GB, no re-download of the 111 GB:

1. 1.keep unsloth's shards 2, 3 and 4 and rename them -0000N-of-00005.gguf
2. 2.drop in …-00001-of-00005.gguf (their shard 1 with the metadata changed: block_count 49, nextn_predict_layers 1, compress_ratios extended, split.count 5) and …-00005-of-00005.gguf (the 29 head tensors, the same bytes as the Q4KM head above)
3. 3.run with --spec-type draft-mtp and no -md; the draft context is created against the target and shares its memory (~0.5 GB less than the separate head)

Made with `patches/merge-mtp-shard.py`, which does the same to any split qwen4exp GGUF given a head file. Measured: within a few percent of the -md form (34.5 vs 36.7 t/s on code, same on prose) -- the convenience is the point, not speed.

Measured (AMD Strix Halo / Radeon 8060S, 128 GB unified, temperature 0, 300 tokens)

Where the remaining time goes on this GPU, profiled: a target pass is ~47 ms for one token and only ~4.4 ms per extra verified token, a head step ~3.4 ms, and about a third of each pass is kernel-launch gaps (the target graph is ~8,000 nodes). Vulkan launches cheaper, which is the whole of its lead.

How it was made

# the 31 bf16 shards that hold mtp.* plus embedtokens / lmhead / the PLE constants (58 GB) python converthftogguf.py --mtp --outtype bf16 --outfile Qwen3.8-Flash-Next-MTP-BF16.gguf src/ llama-quantize --output-tensor-type Q4K --token-embedding-type Q4K \ Qwen3.8-Flash-Next-MTP-BF16.gguf Qwen3.8-Flash-Next-MTP-Q4KM.gguf Q4K_M

34 tensors: blk.48.* (attention, indexer, MoE, hyper-connections), blk.48.nextn.{eh_proj,enorm,hnorm}, token_embd, output, output_hc_*; nextn_predict_layers = 1, block_count = 49.

Credits

qwen4exp support: Daniel Han (PR 27742). MTP graph: JJJYmmm (PR 27739), reconciled and the converter export by LaurentZuijdwijk. The "match the target's quant" observation: agentionai. Base model: the Qwen team, Qwen Community License 1.0 (included).