analogalok/Qwen3.8-27B-DFlash2-Q2_K-GGUF

Qwen3.8-27B-DFlash2-Q2_K (2-Bit Zero-Loss Speculative Drafter)

This repository provides the `Q2_K` (2-bit) quantization of z-lab/Qwen3.8-27B-DFlash2 (~700 MB on disk).

It is engineered specifically for 24GB consumer GPUs (NVIDIA GeForce RTX 4090 / RTX 3090) to run speculative decoding alongside Qwen3.8-27B dense models without spilling into system RAM.

🚀 Benchmark Discovery: Zero Quality Loss at 2-Bit

By quantizing the DFlash 2 block-diffusion drafter down from the official Q4_K_M (~1.1 GB) to Q2_K (~700 MB), we save ~400–450 MB of VRAM with zero degradation in speculative draft accuracy.

Strict A/B Performance on Single RTX 4090 (24GB VRAM):

Evaluated on `Qwen3.8-27B-UD-Q4_K_XL` using `llama-server` (PR #27342) with `--spec-draft-n-max 3` across a 28k prompt baseline.

Key Takeaways:

1. 1.Identical Acceptance Rate: The target 27B model accepts 2.81 out of 3.0 drafted tokens on average $(93.7\% \text{ theoretical efficiency})$, matching the 4-bit drafter identically.
2. 2.Context Expansion: The ~400 MB saved VRAM directly unlocks:
3. 3.170,000 Context on Q4_0 KV (~76 t/s @ 23.89 GB)
4. 4.100,000 Context on high-precision Q8_0 KV (~75 t/s @ 23.88 GB)

⚙️ Why Q2_K is the True Physical Floor for DFlash 2

DFlash 2 is a cross-layer block-diffusion drafter that extracts hidden states from layers [6, 20, 34, 48, 62] of the base model. Because it is not a standalone causal LM, standalone llama-imatrix calibration fails (dflash requires ctx_other to be set).

This means importance-matrix quants (IQ1_S, IQ2_XXS) cannot be computed, making `Q2_K` the absolute lowest non-degrading quantization floor for DFlash 2.

🛠️ Quickstart with llama.cpp

Requires llama.cpp built from PR #27342.

1. Build llama.cpp:

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git fetch origin pull/27342/head:pr-27342
git switch pr-27342

# Build with CUDA
cmake -B build -DGGML_CUDA=ON && cmake --build build -j