HermiHg/Qwen3.8-27B-DFlash2-Q2_K_S-MIX-GGUF

Qwen3.8-27B-DFlash2 — Q2KS-MIX draft (half the reference size)

A mixed-precision 2–3-bit quantization of the DFlash 2 draft model for `Qwen/Qwen3.8-27B`, built to be 50% the size of the reference Q4_K_M checkpoint while retaining ~98% of its throughput.

Q2_K_S-MIX is a mixed-precision quant that compresses the large feed-forward blocks hard while keeping the small, high-impact tensors (the path selector and feature projection) more precise, so it lands at half the reference size with only a small acceptance loss.

Measured performance (vs the reference Q4_K_M)

All draft models served with llama-server (DFlash 2, PR #27342) against the Qwen3.8-27B target on a 24 GB NVIDIA GPU, one fixed conversational prompt with medium reasoning effort, temperature 1.0, concurrency 1, 5 replicate runs. n_max is the draft block length (speculative tokens drafted per verification step).

Q2_K_S-MIX is the smallest draft (535 MiB) and sits on the size–throughput frontier: it posts the lowest acceptance of the four but converts it into within-a-few-percent throughput at every n_max. Non-imatrix Q2K quants were strictly worse than `Q2K_S-MIX` — larger, with lower acceptance and throughput — so they are omitted here.

The 2.33-bit draft model accepts slightly fewer tokens per step than the 4.76-bit reference (e.g. 0.524 vs 0.539 acceptance, 2.57 vs 2.62 draft len at n_max=3), which shows up as a small throughput gap (within a few percent at each n_max). Because DFlash 2 is lossless, this costs speed, not quality — for the same prompt the output is accepted by the same target at the same quality; the smaller drafter just needs marginally more verification steps.

If you have the VRAM to spare, I highly recommmend the `Q3_K_M imatrix quant` as an option — it posts higher acceptance at every n_max at the the cost of just a bit of context size when memory-constrained, however vision is broken on that release until you run the patcher script in this repo to add the required metadata.

Throughput vs draft size, by block length n_max:

<div align="center"> <img src="assets/tgvsmib.png" alt="Throughput vs draft size for n_max = 3, 4, 5" width="100%"> </div>

How it was built (changes vs the reference)

Built clean from the upstream BF16 drafter (incoai/Qwen3.8-27B-DFlash2 GGUF) with `llama-quantize` on a build with DFlash 2 support (PR #27342). No dequant-from-quant: the source is the full-precision checkpoint. It is a q2_k_s base with per-tensor --tensor-type overrides, and the low-bit I-quants are quantized with a real activation-calibrated importance matrix (imatrix) — captured from the draft's own decode activations — rather than a flat one, so each super-block is weighted by the magnitudes it actually sees:

The feed-forward block is 69% of the parameters, so it carries the size savings; the selector and fc are kept higher-precision because they drive which tokens the draft model proposes (acceptance), and the norms/conv-bases stay full precision.

Required build: DFlash 2 support

This drafter needs a llama.cpp build with DFlash 2 support (merged into main on 2026.08.27).

Saving memory elsewhere: multimodal projector (mmproj)

If you are also looking to save memory on the vision side, try my `Qwen3.8-27B-mmproj-Q5_K-MIX` projector checkpoint: 37% of the size of the upstream BF16 mmproj (331 MiB vs 888 MiB) at a measured accuracy cost within the evaluation's noise (see its model card for the full numbers).

Usage

Install llama.cpp with DFlash 2 support, then serve with this checkpoint as the draft:

llama-server \
  -hf <your-target-repo>/Qwen3.8-27B-GGUF:<target-file> \
  -hfd HermiHg/Qwen3.8-27B-DFlash2-Q2_K_S-MIX-GGUF:Q2_K_S \
  --spec-type draft-dflash \
  --spec-draft-n-max 3

Qwen3.8-27B-DFlash2-GGUF

Blog | GitHub

This repository contains GGUF conversions of `incoai/Qwen3.8-27B-DFlash2`, the DFlash 2 draft model for `Qwen/Qwen3.8-27B`. It is not a standalone language model: it runs inside a speculative decoding server and drafts tokens for the target model to verify. The checkpoints are also mirrored at `z-lab/Qwen3.8-27B-DFlash2-GGUF`.

DFlash 2 is a block-diffusion drafter for speculative decoding. It predicts a whole block of tokens in a single pass and keeps the top candidates at every position. A lightweight selector then traces one coherent path through them. Two-tap dynamic convolutions in the backbone keep the draft from decaying toward the end of the block. Decoding is lossless: greedy output matches the target model exactly, and sampling preserves its distribution.

<div align="center"> <img src="assets/dflash2-figure.png" alt="DFlash 2: parallel block drafting with a candidate path selector" width="100%"> </div>

Quick Start

Build llama.cpp with DFlash 2 support (PR #27342):

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git fetch origin pull/27342/head:pr-27342
git switch pr-27342

# NVIDIA CUDA
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
cmake --build build -j

# Apple Silicon
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON
cmake --build build -j

Then serve:

./build/bin/llama-server \
  -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M \
  -hfd incoai/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M \
  --spec-type draft-dflash \
  --spec-draft-n-max 7

See the blog post for other engines and more details.

Evaluation

• —Target: `ggml-org/Qwen3.8-27B-GGUF`, Q4_K_M
• —Sampling: Qwen3.8's officially recommended parameters (temperature 1.0, top-p 0.95, top-k 20), with xhigh reasoning effort
• —Maximum new tokens: 2048
• —Prompts: the first eight GSM8K test examples

Acceptance Length

Acceptance length is the per-request mean of completion tokens divided by verification steps. Higher is better.

Full evaluations of the base checkpoint are on the main model card.

Citation

If you find DFlash 2 useful, please cite:

@misc{inco2026dflash2,
  title  = {{DFlash 2: Keep Drafting Parallel}},
  author = {{Inco AI}},
  year   = {2026},
  month  = {August},
  url    = {https://inco.ai/blog/dflash2/}
}

Please also cite the original DFlash paper:

@inproceedings{chen2026dflash,
  title     = {{DFlash: Block Diffusion for Flash Speculative Decoding}},
  author    = {Chen, Jian and Liang, Yesheng and Liu, Zhijian},
  booktitle = {International Conference on Machine Learning (ICML)},
  year      = {2026}
}