Modelpublic

cruizba/ISTA-DASLab-Qwen3.8-27B-GSQ-RCO-GGUF-Unsloth-MTP

sourceHugging Faceapache-2.0updated 13d agoView on Hugging Face

18likes14kdownloads

Model Card

ISTA-DASLab Qwen3.8-27B GSQ-RCO + native MTP head (grafted, unofficial)

[!NOTE] ⚠️⚠️ This repo is obsolete. ⚠️⚠️ ISTA-DASLab now ships official GSQ-RCO quants with the native MTP head already baked in. Do not use these grafted files — switch to the official -mtp builds and treat this repo as archived. Official MTP GGUFs: - Repo: ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF - IQ2_XS-mtp - IQ2_S-mtp - IQ3_XXS-mtp - IQ3_S-mtp

These are ISTA-DASLab's GSQ-RCO quants of Qwen3.8-27B (IQ2_S, IQ2_XS, IQ3_XXS) with the model's native MTP head grafted back in, so llama.cpp can use multi-token speculative decoding (--spec-type draft-mtp) for a large decode speed-up. This is an unofficial community repo — the quants are ISTA-DASLab's work and the head is borrowed from unsloth; only the graft is added here. It is also a stopgap: the day ISTA-DASLab ships official MTP-integrated quants, this repo loses its purpose — switch to those and archive this one.

The official GSQ files ship without the MTP head; unsloth's UD quants keep it. The head from unsloth's UD-IQ3_XXS (the smallest one that has it) was grafted into all three files — it adds ~195 MiB and nothing else changes: without the MTP flags the files behave exactly like the original quants.

Files

File	What it is	Size	SHA-256
`Qwen3.8-27B-GSQ-RCO-IQ2_S+MTP.gguf`	GSQ-RCO IQ2_S + grafted MTP head (866 tensors)	9.46 GB	`1e652e47…edbca8f`
`Qwen3.8-27B-GSQ-RCO-IQ2_XS+MTP.gguf`	GSQ-RCO IQ2_XS + grafted MTP head (866 tensors)	8.63 GB	`23584659…8de8161`
`Qwen3.8-27B-GSQ-RCO-IQ3_XXS+MTP.gguf`	GSQ-RCO IQ3_XXS + grafted MTP head (866 tensors)	10.30 GB	`ce61f4c6…c655ec6`
`mmproj-F16.gguf`	Vision encoder (from unsloth, unchanged)	0.93 GB	`cbb841a9…2a0b4e43`
`graft_mtp.py`	The graft script (gguf-py + numpy, no GPU needed)	12 KB	—

Full SHA-256 values:

1e652e4778cff7687c3de9217221f7b49424117166b9644d1cb637560edbca8f  Qwen3.8-27B-GSQ-RCO-IQ2_S+MTP.gguf
235846593f2d5c66d712936057315630d19ddc39b90810b2c4d6d33ab9de8161  Qwen3.8-27B-GSQ-RCO-IQ2_XS+MTP.gguf
ce61f4c6183b928ee7fcfaec82b0e4652b0b77843b1979daa2e610807c655ec6  Qwen3.8-27B-GSQ-RCO-IQ3_XXS+MTP.gguf
cbb841a9ee0636b2ec172f5bb8df2ea8dfeb01e90fe7c6126581d662a0b4e43e  mmproj-F16.gguf

Reproduce the grafts

Only pip install gguf numpy is required (no GPU). Put the target quants (from ISTA-DASLab) and one head donor (from unsloth) in the same folder, then run exactly what generated the files in this repo:

bash

python3 graft_mtp.py Qwen3.8-27B-GSQ-RCO-IQ2_S.gguf   Qwen3.8-27B-UD-IQ3_XXS.gguf   Qwen3.8-27B-GSQ-RCO-IQ2_S+MTP.gguf
python3 graft_mtp.py Qwen3.8-27B-GSQ-RCO-IQ2_XS.gguf  Qwen3.8-27B-UD-IQ3_XXS.gguf   Qwen3.8-27B-GSQ-RCO-IQ2_XS+MTP.gguf
python3 graft_mtp.py Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf Qwen3.8-27B-UD-IQ3_XXS.gguf   Qwen3.8-27B-GSQ-RCO-IQ3_XXS+MTP.gguf

Usage pattern: graft_mtp.py <quant-without-head>.gguf <unsloth-UD-donor>.gguf <output>.gguf. The script copies the 15 head tensors from the donor, sets nextn_predict_layers = 1 and bumps block_count to 65 (without it llama.cpp can't find the head), then re-reads the result and verifies every tensor byte-for-byte before exiting. It works for any quant of this model that ships without the head — e.g. the jrell IQ4_XS files.

Quality: the model does not change

—Every tensor of the original quant is preserved byte-for-byte; measured perplexity is identical original vs grafted (6.1509 = 6.1509 and 5.9673 = 5.9673, same value to the last digit).
—With temperature > 0 (the common case), llama.cpp only keeps a drafted token if it equals what the model would have sampled — so the output distribution is mathematically identical. MTP changes speed, not what the model says.
—With temp 0 (greedy) an occasional near-tie may be worded differently; facts stay the same.
—One known quirk: with raw /completion at temp 1.0, IQ2S sometimes samples EOS as its first token (never observed through the chat template). If you serve raw completions, consider a small `minp`.

Benchmarks (RTX 5070 Ti 16 GB)

All rows measured on this card, b10689, threads 14, ubatch 256, -fa on, KV q40, no mmproj loaded (subtract ~170 MiB peak for real vision requests). **Greedy** = 8K prompt + 128-token decode (temp 0). **E2E** = raw `/completion`, 800-token generation, `ignoreeos`, seed 42, temp 1.0 / top-p 0.95 / top-k 20, ctx 163840. Every row compares the same file with MTP on vs off — off behaves bit-for-bit like the original GSQ quant.

Model	Size	Max ctx (16 GB)¹	PPL²	Greedy tok/s (on / off)	e2e tok/s (on / off)	e2e accept	e2e gain
GSQ IQ2_XS+MTP (this repo)	8.63 GB	224K (216K comfy)	6.1509	88.9 / 47.7	74.8 / 60.7	0.450	+23%
GSQ IQ2_S+MTP (this repo)	9.46 GB	204K (192K comfy)	5.9673	92.0 / 46.2	78.1 / 57.8	0.511	+35%
GSQ IQ3_XXS+MTP (this repo)	10.30 GB	160K	5.8805	92.8 / 45.0³	73.9 / 55.9	0.459	+32%

¹ Max context that loaded and passed the 8K-prefill smoke on this 16 GB card (comfy cap with vision headroom in parens). IQ3XXS+MTP fails at load at ≥192K. ² wikitext-2 chunk perplexity (ctx 2048, identical file & flags for every row — relative values directly comparable; absolute scale differs from full-test-set runs). Grafted files score exactly the same as their originals. ³ IQ3XXS spec-off from the b10680 neutrality run; all other greedy numbers from the b10689 sweep.

—Greedy overstates the gain — under real sampling the draft accept rate is 0.45–0.54 (vs 0.69–0.77 greedy), which is why the e2e gains (+23…+35%) are smaller than the greedy ones (+85…+106%). Always quote the sampling protocol. In the agent chat path (llama-swap, xhigh thinking, warm prefix) IQ3_XXS+MTP measured 78.1 vs 55.2 = +41% (b10680) — endpoint- and prompt-dependent (65.3 tok/s on a random-word body; at depth, 60K-token cached prefix: 42.4 vs 21.97 tok/s spec-off).
—For reference, the unsloth equivalents measured in the same session/protocol: their UD-Q2KXL with native MTP is the fastest file overall (84.3 tok/s, verified to 262K ctx on this card) but its PPL (6.0467) is worse than every GSQ quant here; their UD-IQ3XXS with native MTP caps at 96K ctx on 16 GB (75.4 tok/s, OOM at 128K — 11.9 GB of weights leave no room for the draft plus a useful context, exactly the gap the 10.3 GB GSQ graft fills); and their UD-IQ2S ships without the head, so it cannot run MTP at all (61.5 tok/s). Among files of equal quality, GSQ IQ2_S+MTP is the sweet spot — best PPL of the table with a +35% gain.
—MTP is speed, not space: the draft costs about 1 GB of VRAM; the context caps above already include that cost.

Run it (16 GB VRAM example)

bash

llama-server \
  -m Qwen3.8-27B-GSQ-RCO-IQ3_XXS+MTP.gguf \
  --mmproj mmproj-F16.gguf --no-mmproj-offload \
  --ctx-size 163840 --threads 14 -np 1 -ngl 99 -fa on \
  -ctk q4_0 -ctv q4_0 \
  --batch-size 512 --ubatch-size 256 \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --jinja --chat-template-kwargs '{"reasoning_effort":"xhigh"}'

The only two flags that activate MTP are `--spec-type draft-mtp` and `--spec-draft-n-max 3` — everything else is the 16 GB fit and sampling config. Use `n-max 3`: it measured faster than 2 on these files (93.8 vs 90.0 tok/s @160K). Without --spec-type the file runs exactly like the original quant. On a 16 GB card mind the context caps from the table (160K for IQ3XXS, 192K for IQ2S, 216K for IQ2_XS) — the draft costs about 1 GB of VRAM.

Credits & license

—Quant: ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF (Apache-2.0)
—MTP head donor: unsloth/Qwen3.8-27B-GGUF (Apache-2.0)
—Base model: Qwen/Qwen3.8-27B
—Discussion where this came from: ISTA-DASLab discussion #4

Licensed Apache-2.0, inherited from both source repos.