andrew-paul/Qwen3.8-27B-DFlash2-Q3_K_M-GGUF

Qwen3.8-27B-DFlash2 — imatrix-calibrated GGUF drafts

Two quantizations of the DFlash 2 draft model for `Qwen/Qwen3.8-27B`, both calibrated with an importance matrix captured from the drafter's own activations during live speculative decoding.

This is not a standalone language model. It runs inside a speculative-decoding server and drafts tokens for a target model to verify. On its own it produces nothing useful.

Why these exist

The upstream repositories publish BF16, Q80 and Q4KM, none calibrated. Calibrating a DFlash 2 drafter is not straightforward: `llama-imatrix` cannot run one at all, because the architecture has no output layer and no token embedding and borrows both from its target, so it cannot form a context alone (`dflash requires ctxother to be set`). The collector has to be attached to the draft context inside a running speculative session instead.

Two filters in llama.cpp's collector then discard most of what matters. It skips any batch with fewer than 16 rows — and DFlash 2's draft step evaluates only n_max + 1 tokens, so every draft-decode matmul is rejected — and it only collects tensors named blk.*, which excludes DFlash 2's own top-level fc.weight and selector_hidden.weight. Attaching the stock collector captures 10 of 49 tensors. Relaxing both filters captures 49 of 49.

That difference is the whole point of these files. A patch is included as dflash-imatrix.patch so the calibration is reproducible.

Measurements

One machine, one prompt set, one target, all arms identical apart from the drafter: AMD RX 7800 XT, Vulkan, llama.cpp 1deefcca, a quantized Qwen3.8-27B target, --spec-draft-n-max 4, 16k context, 20 fixed prompts, temperature 0, seed 42, 256-token cap, one slot.

The calibrated Q3 matches the upstream Q4_K_M — 65.55% against 65.56% — at 226 MB less. Calibration is worth +0.35 points at 3 bits and +2.97 points at 2 bits, which is where it really pays: an uncalibrated Q2_K gives up 4.45 points against Q3, a calibrated one only 1.47.

Acceptance is deterministic in this stack and reproduces exactly on repeat runs, which is why it is the only figure reported here. Tokens per second is not, because for a target this size it is governed by how much of your setup fits in VRAM rather than by the drafter — measure it on your own machine, at your own context length.

If you need something smaller still

`HermiHg/Qwen3.8-27B-DFlash2-Q2_K_S-MIX-GGUF` is a mixed 2–3-bit build at 561 MB — another 144 MB below the Q2K here. Measured under the same protocol on the same target it reaches **62.50%**, so it trades 1.58 points of acceptance for that space. If you are tight enough on memory that 144 MB matters, it is the better choice, and it comfortably beats an uncalibrated Q2K at any size.

Provenance

Quantized from Qwen3.8-27B-DFlash2-BF16.gguf (3,860,293,152 bytes) as published in `z-lab/Qwen3.8-27B-DFlash2-GGUF`, itself a conversion of `incoai/Qwen3.8-27B-DFlash2`.

llama.cpp revision 1deefcca395743049c3820ab8f9b15043f3e9446 plus dflash-imatrix.patch. The imatrix was captured from a live speculative session against a quantized Qwen3.8-27B target, with both collector filters relaxed so draft-decode batches and DFlash 2's top-level weights are included.

Pipeline control. Rebuilding Q4_K_M from the same BF16 with an unpatched build yields a file byte-identical to the published one:

18a380efc9b7ed8d88677fc895f5c11ae170653434ee378f7348f715c14d0594

matching z-lab's Qwen3.8-27B-DFlash2-Q4_K_M.gguf exactly. These builds therefore come from a pipeline demonstrably able to reproduce the upstream artifacts.

Usage

llama-server -m <your-Qwen3.8-27B-target>.gguf \
    --spec-type draft-dflash \
    -md Qwen3.8-27B-DFlash2-Q3_K_M.gguf \
    --spec-draft-n-max 4 \
    --spec-draft-ngl 99

n=4 and n=5 measured as statistically indistinguishable; n=6 costs acceptance and gains nothing for it.

Requires a llama.cpp build with DFlash 2 support. The patch is only needed to reproduce the calibration — the GGUFs themselves load on any DFlash 2 build.

Note DFlash 2 carries a few GB of overhead beyond the drafter's own weights, from hidden-state taps and verification buffers, which is comfortable on a large-memory machine and can be prohibitive on a 16 GB GPU.

Credit

DFlash 2 is the work of the z-lab / inco.ai authors (blog, code). This repository redistributes re-quantized copies of their released drafter and claims no part of the method, the training, or the original model. Apache-2.0, inherited from upstream.