mrchuy/Qwen3.8-27B-DFlash-drafter-bootstrap-GGUF

Qwen3.8-27B DFlash Bootstrap GGUF

A Qwen3.8 DFlash drafter transplanted from Qwen3.6 that reaches over 50% first-token acceptance

This is an experimental DFlash speculative drafter for Qwen/Qwen3.8-27B, created by bootstrapping the published Qwen3.6-27B-DFlash weights against the Qwen3.8 tokenizer and target-model metadata.

Despite no additional training on Qwen3.8, the resulting drafter shows substantial speculative transfer:

57.2% first speculative-token acceptance 34.0% overall draft-token acceptance 31.94 tok/s — 34.6% faster than plain Qwen3.8 in this test

The DFlash weights are still the original Qwen3.6-trained weights. The conversion changes target metadata/tokenizer compatibility, but does not retrain the drafter for Qwen3.8.

For comparison on the same target, hardware, and prompt:

Plain Qwen3.8       23.73 tok/s
DFlash bootstrap    31.94 tok/s
Native MTP3         50.70 tok/s

This is not a native Qwen3.8-trained DFlash model. It is a transfer/bootstrap experiment and a possible starting point for a future Qwen3.8-specific DFlash fine-tune.

Controlled benchmark

Same 5,163-token prompt for all runs: Qwen3.8-27B-UD-Q4_K_XL.gguf, 2× RTX 4060 Ti 16 GB, tensor split 1,1, 75K context, 4096/1024 batch, parallel 1, temperature 1.0, top-k 20, and top-p 0.95. Speculative runs use n_max=3.

The bootstrap DFlash is already a measurable speedup over plain inference despite receiving no Qwen3.8-specific training. Native Qwen3.8 MTP remains substantially stronger.

These are single completed generations using stochastic sampling, not a multi-run statistical benchmark.

Plain Qwen3.8 baseline

prompt:  1008.97 tok/s  (5,163 tokens)
decode:    23.73 tok/s  (25,974 tokens)

DFlash bootstrap result

Completed n_max=3 run:

prompt:   783.93 tok/s  (5,163 tokens)
decode:    31.94 tok/s  (21,202 tokens)

avg tokens/verify = 2.02
acc per pos       = (0.572, 0.297, 0.152)

Speculative statistics:

draft calls      = 10,490
draft tokens     = 31,470
accepted tokens  = 10,713
overall accept   = 34.04%

Native Qwen3.8 MTP result

Completed n_max=3 run:

prompt:   959.32 tok/s  (5,163 tokens)
decode:    50.70 tok/s  (22,852 tokens)

avg tokens/verify = 3.15
acc per pos       = (0.855, 0.710, 0.585)

Speculative statistics:

draft calls      = 7,256
draft tokens     = 21,768
accepted tokens  = 15,596
overall accept   = 71.65%

Draft-length experiments

* Interrupted runs; throughput is provisional and no final acceptance statistics were recorded.

† The earlier n_max=4 run used inherited/tensor-parallel draft placement rather than the later single-GPU pinned topology, so its acceptance curve is useful but its throughput is not directly comparable.

The completed runs show a very similar acceptance curve:

              n=3       n=4

position 1    57.2%     55.1%
position 2    29.7%     29.3%
position 3    15.2%     15.0%
position 4       —       7.8%

For the current bootstrap model, n_max=2 or n_max=3 appears to be the useful operating region.

llama.cpp configurations

1. Plain Qwen3.8

[Qwen38-Q4-PLAIN]
hf-repo = unsloth/Qwen3.8-27B-GGUF
hf-file = Qwen3.8-27B-UD-Q4_K_XL.gguf

split-mode = tensor
tensor-split = 1,1

ctx-size = 75000

batch-size = 4096
ubatch-size = 1024
flash-attn = on
parallel = 1

2. Native Qwen3.8 MTP3

[Qwen38-Q4-MTP3]
hf-repo = unsloth/Qwen3.8-27B-GGUF
hf-file = Qwen3.8-27B-UD-Q4_K_XL.gguf

split-mode = tensor
tensor-split = 1,1

ctx-size = 75000

batch-size = 4096
ubatch-size = 1024
flash-attn = on
parallel = 1

spec-type = draft-mtp
spec-draft-n-max = 3

3. Qwen3.8 DFlash Bootstrap

[Qwen38-Q4-DFlash3-bootstrap]
hf-repo = unsloth/Qwen3.8-27B-GGUF
hf-file = Qwen3.8-27B-UD-Q4_K_XL.gguf

split-mode = tensor
tensor-split = 1,1

ctx-size = 75000

batch-size = 4096
ubatch-size = 1024
flash-attn = on
parallel = 1

spec-type = draft-dflash
spec-draft-model = /path/Qwen3.8-27B-DFlash-bootstrap-Q8_0.gguf
spec-draft-n-max = 3
spec-draft-n-min = 0

device-draft = CUDA0
n-gpu-layers-draft = all

llama.cpp compatibility

The default device-placement path works without PR #26636: with a tensor-parallel target, the DFlash drafter can inherit the same tensor-parallel placement.

The benchmarks above instead pin the drafter entirely to CUDA0:

device-draft = CUDA0
n-gpu-layers-draft = all

Using a tensor-parallel target + single-GPU drafter required the shared-backend fix from llama.cpp PR #26636.

Benchmark build:

llama.cpp build 10457
commit 5ad147554

PR #26636 is therefore only required for this pinned-draft topology, not for using the DFlash GGUF generally.

Model details

The resulting draft GGUF contains:

architecture        = dflash
parameters          = 1.73B
block_count         = 5
block_size          = 16
embedding_length    = 5120
feed_forward_length = 17408

attention heads     = 32
KV heads            = 8
head dimension      = 128

target layers       = [2, 17, 32, 47, 62]
sliding window      = 2048

The Q8_0 model is approximately:

1.71 GiB
8.50 BPW

Provenance

Original DFlash weights:

z-lab/Qwen3.6-27B-DFlash

Qwen3.8 target metadata/tokenizer:

Qwen/Qwen3.8-27B

Benchmark target:

unsloth/Qwen3.8-27B-GGUF
Qwen3.8-27B-UD-Q4_K_XL.gguf

No Qwen3.8-specific DFlash training was performed.

Re-create the model

1. Download the original Qwen3.6 DFlash weights

mkdir -p /data/drafters/Qwen3.6-27B-DFlash

hf download z-lab/Qwen3.6-27B-DFlash \
  --local-dir /data/drafters/Qwen3.6-27B-DFlash

2. Download Qwen3.8 metadata and tokenizer

mkdir -p /data/drafters/Qwen3.8-27B-target-meta

hf download Qwen/Qwen3.8-27B \
  config.json \
  tokenizer.json \
  tokenizer_config.json \
  generation_config.json \
  --local-dir /data/drafters/Qwen3.8-27B-target-meta

3. Convert the DFlash model against Qwen3.8

From a llama.cpp checkout:

python convert_hf_to_gguf.py \
  /data/drafters/Qwen3.6-27B-DFlash \
  --target-model-dir /data/drafters/Qwen3.8-27B-target-meta \
  --outtype bf16 \
  --outfile Qwen3.8-27B-DFlash-bootstrap-bf16.gguf

The resulting GGUF should report:

dflash.block_count       = 5
dflash.block_size        = 16
dflash.target_layers     = [2, 17, 32, 47, 62]
dflash.embedding_length  = 5120

4. Quantize to Q8_0

If llama-quantize is not already built:

cmake -B build \
  -DGGML_NATIVE=ON \
  -DLLAMA_BUILD_TESTS=OFF \
  .

cmake --build build --target llama-quantize -j"$(nproc)"

Then quantize:

./build/bin/llama-quantize \
  Qwen3.8-27B-DFlash-bootstrap-bf16.gguf \
  Qwen3.8-27B-DFlash-bootstrap-Q8_0.gguf \
  Q8_0

Expected size:

~1.71 GiB

Status

Experimental, but demonstrably functional.

Without any Qwen3.8-specific DFlash training:

Plain       23.73 tok/s
DFlash3     31.94 tok/s   (+34.6%)
MTP3        50.70 tok/s   (+113.7%)

The transplanted drafter reaches 57.2% first speculative-token acceptance and 34.04% overall acceptance, demonstrating both cross-version speculative transfer and a measurable speed improvement over plain inference.

Native Qwen3.8 MTP remains substantially stronger. A natural next step is fine-tuning the transplanted DFlash weights directly against Qwen3.8 hidden states and outputs.