williamliao/dspark_gemma4_12b-GGUF

Gemma 4 12B DSpark GGUF

GGUF conversion of the DeepSeek DSpark draft model for `google/gemma-4-12B-it` (deepseek-ai/dspark_gemma4_12b_block7, block size 7).

This is a draft model only. It cannot be used by itself. Pair it with a compatible Gemma 4 12B target model and run it with DSpark speculative decoding in llama.cpp.

[!IMPORTANT] Experimental, and not directly comparable to other DSpark cards in this collection. This model uses llama.cpp PR #25549 — the native Gemma 4 DSpark implementation — not PR #26275 (speculators-format checkpoint support) used for the 26B-A4B and 31B DSpark cards. Different PR, different upstream checkpoint, different block size, and this benchmark also enables confidence gating (--spec-draft-p-min 0.60) that the other cards do not use. Don't read the acceptance/speedup numbers below as an apples-to-apples architecture comparison against those cards — the delta could easily come from any of those four variables, not from target size.

Files

Recommended target: google/gemma-4-12B-it (may be quantized). Draft and target must use the same tokenizer/model family.

Conversion

hf download google/gemma-4-12B-it --local-dir Gemma4-12B-it
hf download deepseek-ai/dspark_gemma4_12b_block7 --local-dir dspark_gemma4_12b_block7

python convert_hf_to_gguf.py dspark_gemma4_12b_block7 \
  --outtype bf16 \
  --target-model-dir Gemma4-12B-it \
  --outfile gemma4_12b-DSpark-BF16.gguf

llama-quantize gemma4_12b-DSpark-BF16.gguf gemma4_12b-DSpark-Q4_K_M.gguf Q4_K_M

--target-model-dir must point at the target checkpoint (Gemma4-12B-it), not the DSpark directory — the draft checkpoint doesn't ship its own tokenizer.

llama.cpp usage

llama-server.exe ^
  -m "gemma-4-12B-it-Q4_K_M.gguf" ^
  --model-draft "gemma4_12b-DSpark-Q4_K_M.gguf" ^
  --host 0.0.0.0 ^
  --port 18010 ^
  --device-draft CUDA1 ^
  --threads 12 ^
  --threads-batch 12 ^
  --split-mode layer ^
  --tensor-split 14,12 ^
  --ctx-size 65536 ^
  --no-mmap ^
  -ngl -1 ^
  --flash-attn on ^
  --ubatch-size 256 ^
  --batch-size 2048 ^
  --cache-type-k q8_0 ^
  --cache-type-v q8_0 ^
  --temp 1.0 ^
  --top-k 64 ^
  --top-p 0.95 ^
  --reasoning off ^
  --fit off ^
  --parallel 1 ^
  --spec-type draft-dspark ^
  --spec-draft-n-max 7 ^
  --spec-draft-p-min 0.60

--spec-draft-n-max 7 matches the checkpoint's block size 7. --spec-draft-p-min 0.60 gates low-confidence proposals before verification — see Interpretation for why this matters for the acceptance numbers below.

Benchmark

Setup: RTX 5070 Ti 16GB + RTX 5060 Ti 16GB, target Gemma 4 12B-it Q4KM, context 65,536, temp 1.0/top-k 64/top-p 0.95, reasoning off, 1 parallel slot, mtp-bench.py, mostly 192 tokens/test. Draft only tested at Q4KM — no BF16/Q8_0 comparison was run (see Limitations).

Not directly comparable to the BF16 target benchmarks in the upstream PR, or to the 26B-A4B / 31B DSpark cards (different PR/checkpoint/config — see the warning above).

Aggregate

Baseline was stable at 72.5–75.7 tok/s across all tasks regardless of workload — the variation in the DSpark column is entirely from drafting behavior, not target-side variance.

Interpretation

The strongest gains were on predictable, structured output: repeat pattern (2.63×), JSON (2.48×), stepwise math (2.08×), code completion (1.90×), Python (1.67×). Open-ended generation benefited less — creative_short was essentially unchanged, translation gained only ~8%. This matches the pattern seen on the other Gemma 4 DSpark cards: structured/constrained token sequences are easier for a small draft model to predict than open-ended or high-entropy text.

On the 78.9% aggregate acceptance specifically: this is much higher than the 26B-A4B card's 43.56% and the 31B card's 60.82%, but this run also used --spec-draft-p-min 0.60, which the other two didn't. Confidence gating filters out low-confidence draft proposals before they're sent for verification, which mechanically inflates the acceptance ratio (you're only counting the proposals the draft was already confident about). Some of this run's advantage may be genuine — different PR, different checkpoint — but some of it is very plausibly just pmin doing its job. This isn't validated either way without an ablation (same checkpoint, pmin off vs. 0.60), so treat the 78.9% number as "this run's config," not "this checkpoint's ceiling acceptance."

High acceptance still doesn't fully explain the 2.77× end-to-end speedup — draft execution cost, verification batching, quantization, and cross-GPU transfer all factor in too.

Notes

• —DSpark performance depends heavily on workload predictability.
• —High acceptance does not guarantee proportional throughput.
• —The upstream PR recommends keeping the draft in BF16; this benchmark only tested Q4KM, so no local evidence either supports or contradicts that — if quantization sensitivity matters to your deployment, test BF16/Q80/ Q4KM yourself before assuming Q4K_M is sufficient.
• —Gemma 4 DSpark support remains experimental and may change while PR #25549 is under review.

Limitations

• —Draft model cannot generate output standalone.
• —Only Q4KM draft quantization was benchmarked — no BF16/Q8_0 comparison.
• —Results are single-configuration, single-hardware local measurements, not a controlled comparison against the 26B-A4B/31B DSpark cards (different PR, checkpoint, and --spec-draft-p-min setting — see the notice above).
• —Confidence-gating contribution to the acceptance rate has not been isolated via ablation.
• —Vision/multimodal behavior not tested.

Credits

• —Original target: google/gemma-4-12B-it
• —Original DSpark checkpoint: deepseek-ai/dspark_gemma4_12b_block7
• —llama.cpp Gemma 4 DSpark implementation: PR #25549

This repository only provides a GGUF conversion/quantization. Please follow the licenses and usage terms of the original target model, draft model, and llama.cpp.