williamliao/dspark_gemma4_26b-a4b-it-GGUF

Gemma 4 26B-A4B-it DSpark Speculator — GGUF

GGUF conversion of the DSpark speculative-decoding draft model for google/gemma-4-26B-A4B-it.

This repository does not contain a standalone language model. The GGUF file is a draft/speculator model and must be loaded together with a compatible Gemma 4 26B-A4B-it target model.

Update — llama.cpp PR #26275 / e107846

The GGUF files have been reconverted using llama.cpp PR #26275 commit `e107846`, which adds proper support for Speculators/SpecForge-format DSpark checkpoints.

This update is important because these checkpoints use a DFlash-style 1+N bonus-anchor block layout:

• —the first slot is a bonus/anchor token, not a prediction slot;
• —a block of size 7 therefore contains at most 6 actual draft predictions;
• —llama.cpp now records this using dflash.bonus_anchor;
• —DSpark prediction/confidence is read from the mask positions rather than the anchor;
• —reduced draft vocabularies can be expanded correctly by the converter.

Because the GGUF representation itself changed, older conversions should be replaced by GGUF files produced with the updated converter.

Practical result

On the same RTX 5070 Ti + RTX 5060 Ti consumer-GPU system used for the earlier tests, the updated conversion improved DSpark throughput substantially without materially changing draft acceptance.

The main remaining observation is unchanged:

DFlash is still faster overall on this fast MoE target, but the gap is smaller than with the previous DSpark conversion.

Status

DSpark support is being developed in:

ggml-org/llama.cpp#26275

The results below use the PR branch including:

e107846f1963bd24ef7940c20a4ae4f8d03070b9

Until the PR is merged, use a llama.cpp build containing this commit or a newer revision of the PR.

Compatibility

The target model and speculator must use compatible tokenizer and vocabulary metadata.

Recommended settings

speculative type: DSpark
draft max: 5

max 5 remains the best practical configuration in this benchmark.

The updated runtime correctly clamps this checkpoint to an effective maximum of 6 draft tokens. Values larger than 6 therefore do not create additional draft positions.

For example, max 6, max 7, and max 15 all produced exactly:

total draft:          3639
total draft accepted: 1403
aggregate acceptance: 38.55%

The wall-time differences between those runs should therefore be treated as normal run-to-run variance rather than a real benefit from setting a larger n_max.

Benchmark

Test system

Target:

gemma-4-26B-A4B-it-UD-Q4_K_M

Hardware:

RTX 5070 Ti 16GB
RTX 5060 Ti 16GB

Workload:

13 requests
single-stream mixed workload
2035 predicted tokens

Tasks include:

• —Python
• —C++
• —concept explanation
• —summarization
• —factual QA
• —translation
• —creative writing
• —stepwise math
• —JSON
• —long reasoning
• —repeated patterns
• —code completion
• —long code review

Updated DSpark draft-length sweep

After e107846

• —max 6, 7, and 15 execute the same effective draft workload because the checkpoint is capped at six prediction slots. Their wall-time differences are measurement variance and should not be interpreted as draft-depth scaling.

Why max 5 still wins

Going from max 5 to the effective max 6:

additional draft tokens:    +473
additional accepted tokens:  +24

That is a large increase in draft work for only a very small increase in accepted tokens.

For this hardware and target, the sixth draft position does not pay for its execution cost across the mixed workload.

Before vs. after e107846

The old GGUF conversion used the earlier DSpark handling before the SpecForge bonus-anchor layout was represented explicitly.

This is approximately:

108.8 → 121.5 tok/s
≈ +11.7%

while aggregate acceptance remains exactly:

43.56%

This is an interesting result because it suggests that the improvement is primarily related to the corrected conversion/runtime handling rather than improved prediction quality.

The new implementation also prevents configurations such as max 7 from performing an extra invalid/unused prediction position.

Previously:

max 7:
34.09% acceptance
27.31 s
91.4 tok/s

With e107846, values above the trained six prediction positions are correctly capped.

Per-workload — updated DSpark max 5

The workload dependence remains very strong.

Highly predictable outputs such as JSON, repeated patterns, and code completion benefit substantially from speculative decoding.

Translation, creative writing, and long code review continue to have relatively low acceptance and can perform worse than normal target decoding.

DSpark vs. previous DFlash result

The previous benchmark on this system measured:

Using those measurements:

Old DSpark vs baseline: ~1.10x
New DSpark vs baseline: ~1.23x
DFlash vs baseline:     ~1.47x

The DFlash advantage over DSpark therefore decreases from roughly:

145.7 / 108.8 = 1.34x

to approximately:

145.7 / 121.5 = 1.20x

So the corrected DSpark conversion/runtime path meaningfully narrows the gap, but DFlash still leads overall on this hardware.

Important comparison caveat

The baseline and DFlash figures above come from the earlier benchmark revision, while the updated DSpark result uses PR #26275 commit e107846.

They are useful for understanding the approximate trend, but a strict performance comparison should rerun baseline, DSpark, and DFlash on the exact same llama.cpp revision.

Interpretation

Gemma 4 26B-A4B is a fast MoE target.

Only a subset of parameters is active for each token, so target decoding is already relatively inexpensive. This changes the economics of speculative decoding: draft-model overhead becomes a larger fraction of total generation time.

The updated results suggest several things.

1. The SpecForge layout fix matters

The new conversion/runtime path improves max-5 throughput from:

108.8 → 121.5 tok/s

without changing aggregate acceptance.

So at least part of the previous performance loss was related to runtime/conversion handling rather than draft-model prediction quality.

2. Longer drafts are still not automatically better

Max 6 generates about 15% more draft tokens than max 5:

3166 → 3639

but gains only:

1379 → 1403 accepted tokens

For this consumer-GPU setup, that trade is not worthwhile.

3. DSpark performance remains workload-dependent

DSpark performs very well on:

• —JSON
• —repeated patterns
• —code completion
• —Python/C++ generation

but poorly on:

• —translation
• —creative writing
• —long code review

This is consistent with speculative decoding being most effective when upcoming tokens are predictable.

4. DFlash remains faster overall

Even after the SpecForge correction, the previous DFlash result remains substantially faster than the updated DSpark result.

However, the gap is now closer to ~20% rather than ~34%.

This makes the remaining difference especially interesting: it may reflect DSpark runtime overhead, hardware characteristics, checkpoint behavior, or other implementation details rather than simply an incorrect GGUF conversion.

Further apples-to-apples testing on the same llama.cpp revision is needed before assigning the remaining gap to any single cause.

Suggested use

Reproducibility notes

Speculative-decoding performance depends heavily on:

• —target quantization;
• —draft quantization;
• —GPU assignment;
• —PCIe topology;
• —context length;
• —KV-cache type;
• —batch and ubatch size;
• —sampling parameters;
• —draft confidence threshold;
• —draft maximum;
• —prompt/output distribution;
• —llama.cpp revision.

For meaningful comparisons:

1. 1.use the same target GGUF;
2. 2.use identical prompts and stopping conditions;
3. 3.use the same llama.cpp commit;
4. 4.compare total predicted tokens divided by total wall time;
5. 5.record both acceptance and total draft work;
6. 6.repeat runs before trusting small differences.

Aggregate acceptance alone is not enough to predict throughput.

A configuration can have good acceptance while still losing performance if the extra draft computation costs more than the accepted tokens save.

Limitations

• —This GGUF is a speculator only and cannot generate standalone.
• —It does not reduce the VRAM/storage requirement of the target model.
• —Results were measured on consumer NVIDIA GPUs.
• —Performance may differ substantially on RTX PRO, RTX 6000, A100, H100, B200, or other accelerators.
• —DFlash results should be rerun on the same llama.cpp commit for a strict post-e107846 comparison.
• —Translation and creative writing remain poor speculative-decoding workloads in this test.
• —Vision/multimodal speculative decoding was not tested.
• —PR #26275 is still under development at the time of writing.

Credits

• —Target model: Google DeepMind — Gemma 4 26B-A4B-it
• —Original DSpark checkpoint: makora-ai/gemma4-26b-a4b-dspark
• —Runtime integration: ggml-org/llama.cpp PR #26275
• —SpecForge support / bonus-anchor handling: llama.cpp commit e107846
• —GGUF conversion and consumer-GPU benchmarks: williamliao

License

This repository contains a converted speculator checkpoint.

Use is subject to the licenses and terms of the original speculator checkpoint, the Gemma target model, and the relevant software projects. Review all upstream licenses before redistribution or commercial use.