cafkafk/gemma-4-26B-A4B-it-assistant-GGUF-noimatrix

gemma-4-26B-A4B-it-assistant — GGUF — noimatrix (ik_llama.cpp only)

GGUF quantizations of google/gemma-4-26B-A4B-it-assistant, the Multi-Token-Prediction (MTP) drafter for gemma-4-26B-A4B-it. Intended for use as the draft model in speculative decoding paired with a quantized gemma-4-26B-A4B-it verifier. Realised speedup depends on hardware and verifier choice — see §Performance for measured numbers and the PR's reference benchmarks.

[!WARNING] ik_llama.cpp only — will NOT load in mainline llama.cpp These quants use the gemma4_mtp architecture, which is currently only supported in `ikawrakow/ik_llama.cpp` on (or after) the `feat/gemma-4-mtp` branch / PR #1744. Mainline ggml-org/llama.cpp does not know about gemma4_assistant / gemma4_mtp and will refuse to load these files. Downstream tools that wrap mainline (ollama, LM Studio, jan.ai, llama-cpp-python, …) will not work either until they follow ikllama.cpp. Tested with ikllama.cpp commit a703033607ed3edbeab0205d8c9ad75cc1b5759f.

[!NOTE] Benchmarks in progress Real measured throughput + acceptance-rate numbers for these drafters are being collected on a consumer laptop GPU (NVIDIA 4060 Laptop, 8 GiB VRAM) — full --draft-max × --draft-p-min matrix across multiple prompts, driven against llama-server with acceptance read from its per-request statistics mtp: stderr lines. Results land in the smaller drafters' model cards first (E2B → E4B → 26B-A4B → 31B), since smaller models cycle through the bench faster. Check the [E2B card](https://huggingface.co/cafkafk/gemma-4-E2B-it-assistant-GGUF-noimatrix) first if you're shopping for performance numbers — even if you plan to deploy a different size, the relative draft-max curves and acceptance trends carry across sizes within the family. Until then, §Performance below cites the upstream PR's reference benchmarks (data-center GPU + 31B verifier) — treat them as a ceiling, not a target.

[!IMPORTANT] Honest limitations of this build Things that should be on the model card and aren't faked: - No imatrix calibration. PR #1744 builds the gemma4mtp drafter graph with a hardcoded `GGMLASSERT(hastargetctx), meaning standalone-drafter llama-imatrix runs abort in llamadecode` before producing anything. Same as every other community-published Gemma 4 drafter quant today (Radamanthys11's, etc.), this build quantizes from f16 directly. At ≥4 bits this is fine; the precision benefit of imatrix-guided quantization at Q5+ is single-digit %. At Q3 and below it matters more — see the per-quant warnings in the table below. - **No acceptance-rate validation in this build.** The benchmark numbers in §"Performance" come from the upstream PR thread on a 31B verifier, not from runs against *this* drafter quant. Smoke-testing on the build host (CPU-only) was disabled because per-token MTP cost is many seconds on CPU, making proper benchmark-quality runs impractical mid-pipeline. Treat the numbers as expectations, not measurements of these specific files. - **No IK-only IQK quants** (IQ4KS / IQ5KS / IQ4KSS). These are normally what "ikllama.cpp build" gets you over mainline, but their precision-per-bit advantage comes from imatrix-guided scale selection — without imatrix they collapse to roughly K-quant quality at the same bit budget, so shipping them would just be misleading row-count padding. They'll come back in a sibling experiment when upstream supports standalone-drafter imatrix. - **MXFP4 has narrower runtime support.** Loadable in current ikllama.cpp and mainline llama.cpp; older ggml-based runtimes may not support it yet. Use a K-quant if you need to load these in something older.

Pairing — required

A drafter is not a standalone language model. To use these quants you also need a base-model GGUF, with matching vocab (262144 tokens, which is the whole Gemma 4 family default). Recommended pairings:

Verifier: unsloth/gemma-4-26B-A4B-it-GGUF — ~25.2B total (3.8B active, MoE) parameters. Heads-up on Google's naming: the "E" in E2B/E4B means effective (active inference) parameters via Per-Layer Embeddings, not total weight count — full weights still have to fit in VRAM. The "A" in 26B A4B is the same trick for the MoE variant: 3.8B active out of 25.2B total. The 31B is plain dense, no naming games.

Pairing precision: matching is generally optimal, but mismatched pairings work too with some acceptance-rate penalty. Both bf16 and f16 drafters in this repo are valid pairing targets for a bf16/f16/Q8_0 verifier — bf16 is preferred when your runtime supports it (matches the source-tensor format exactly; see the bf16 row in §Quants). Going above the verifier's precision on the drafter has no benefit.

Empirical note (single-data-point, no warranty): on a 4060 Laptop GPU paired against unsloth's Gemma-4-E4B-it-Q4_K_M.gguf verifier, the Q4_K_M drafter outperformed the Q8_0 drafter by ~13% in throughput at --draft-max 3. The smaller drafter's faster draft step appears to outweigh the acceptance-rate cost from more aggressive quantization. Contradicts the common "always pick the highest-bit drafter" heuristic. Bench your own hardware before assuming.

Preserves structured-output tokens

The drafter's vocabulary is identical to the verifier's (262144 tokens, the Gemma 4 family default). Notably, that includes Gemma 4's reserved tokens for structured output formats which the drafter speculates correctly:

If you're running tool-calling agents, multi-step reasoning, or any structured-generation workflow on top of Gemma 4, this drafter will speculate those tokens just like any other — meaning the MTP speedup applies to the whole response, not just the natural-language parts. Most published drafter quants don't talk about this because it Just Works mechanically (vocabulary is a separate GGUF section that's never quantized), but it's worth saying out loud: pairing this drafter with a tool-calling-finetuned verifier preserves the tool-call grammar end-to-end.

Quants

~408M parameter MTP head, 2816-dim backbone projection (must match the verifier's hidden_size).

(GGUF general.size_label for this drafter is the verifier's MoE "A4B" tag, not a head parameter count; ~408M is computed from the bf16 file size, ~816 MB / 2 bytes-per-param.)

12 files spanning bf16 → Q3, in approximate order of decreasing precision.

Deliberately omitted quants (and why, briefly):

• —F32 — zero-padded bf16, no information gain, double the disk.
• —Q4_0 / Q5_0 / Q4_1 / Q5_1 — legacy non-K quants. K-quants strictly dominate them at the same bit budget.
• —Q3_K_M / Q3_K_S / Q2_K — without imatrix, drafter acceptance drops sharply below Q3KL. Re-add when imatrix is available.
• —IQ2_* / IQ1_* — too noisy at any bit budget for drafter use, even with imatrix. Verifier rejects most drafted tokens, paired generation goes net negative vs. baseline.
• —IQ4_KS / IQ4_KSS / IQ5_KS / IQ3_KT / IQ4_KT — IK-fork-only quants whose precision advantage requires imatrix. Coming in a future imatrix-capable sibling experiment.

Usage

ik_llama.cpp's llama-server (or llama-cli for one-shot generation):

# Build / install ik_llama.cpp first; see
# https://github.com/ikawrakow/ik_llama.cpp

llama-server \
    --model gemma-4-26B-A4B-it-Q8_0.gguf \
    --model-draft gemma-4-26B-A4B-it-assistant-Q8_0.gguf \
    --spec-type mtp \
    --draft-max 3 \
    --draft-p-min 0.0 \
    -ngld 99 \
    --n-gpu-layers 99 \
    --ctx-size 32768 \
    -ctk q8_0 -ctv q8_0 \
    -b 1024 -ub 1024 \
    --jinja \
    --host 127.0.0.1 --port 18080

Flag reference:

--spec-autotune (per the PR #1744 description) will probe several --draft-max values during inference and pick the best-fitting one for your workload — useful if you don't want to tune by hand.

Performance

<!-- TODO: bench numbers pending; see cafkafk/gemma-4-E2B-it-assistant-GGUF-noimatrix for reference shape -->

Reproducing the upstream benchmark on a 31B verifier + this drafter at Q80 on Q80 (per the PR #1744 description):

Smaller verifiers (E2B/E4B) get less absolute t/s benefit because the verifier itself is faster, so there's less time-budget for the drafter to fill in. The percentage uplift is similar.

Compatibility notes

A few cosmetic / non-blocking quirks you may see in normal use:

• —`transformers` warning during conversion (only relevant if you re-convert from source rather than using these prebuilts):

  You are using a model of type `gemma4_assistant` to instantiate a
  model of type ``. This may be expected if you are loading a
  checkpoint that shares a subset of the architecture …

The IK fork's convert_hf_to_gguf.py patches in gemma4_assistant arch support on the GGUF side but does not patch the Hugging Face transformers library itself. So transformers (which the converter uses to read the source safetensors) sees the unfamiliar model_type and falls back to generic loading. Generic loading reads the raw weights correctly, so the conversion still produces a valid GGUF — the warning is cosmetic.

• —*`Oops: tensor with strange name per_layer_` at runtime* (visible if you pair against certain non-google-flavored Gemma 4 base GGUFs, e.g. unsloth's). These warnings come from the verifier* loader, not the drafter — they're the verifier model's per-layer projection tensors which ik_llama.cpp's gemma4 base implementation may not fully recognize on third-party-quantized GGUFs. Inference still works but may fall back to slower code paths for those tensors. If absolute throughput seems too low vs. the PR's reference benchmarks, try a different verifier (google's own f16, bartowski's quants, or any other community source) and compare.

• —`mtp_pre_proj.weight` / `mtp_post_proj.weight` "strange name" warnings at drafter load — see PR #1744 review thread; these are the drafter's MTP projection tensors which the size- accounting iteration in src/llama.cpp doesn't special-case. Cosmetic; the MTP runtime loads them correctly via create_gemma4_mtp_tensors.

Provenance

• —Source: google/gemma-4-26B-A4B-it-assistant, Apache 2.0 + Gemma terms of use.
• —Architecture: gemma4_mtp (the GGUF-side name for Gemma4AssistantForCausalLM).
• —Converter / runtime: ikllama.cpp `feat/gemma-4-mtp` branch, i.e. [PR #1744](https://github.com/ikawrakow/ikllama.cpp/pull/1744) by @SamuelOliveirads.
• —Calibration corpus for imatrix: none used in this build (see "Honest limitations" above for why).
• —Build host: a CPU-only Linux box.

Comparable existing community quants: `Radamanthys11/Gemma-4-26B-A4B-it-assistant-GGUF` and the rest of @Radamanthys11's collection (the same person who wrote PR #1744). Those repos ship F16 + Q8_0 only.

This repo ships every quant variant of this drafter that made sense to produce: 12 files spanning bf16 reference down to Q3KL, including K-quants, non-K i-quants (IQ4NL, IQ4XS), and OCP MXFP4. The omitted quants (F32, legacy Q40/Q50 etc., Q2K, IQ2, IQ1_, the imatrix-dependent IQ4_KS family) are documented above the table with the reason each was left out.

License

Gemma Terms of Use, inherited from the source model. By downloading or using these quants you agree to Google's Gemma terms — same as if you'd downloaded the upstream weights directly.

Issues / questions

Open a discussion on this repo (cafkafk/gemma-4-26B-A4B-it-assistant-GGUF-noimatrix) for anything quant-specific (a particular file refusing to load, a quant variant behaving worse than expected, sizes-table corrections, etc.).

For ikllama.cpp runtime bugs (`gemma4mtp arch issues, MTP acceptance-rate quirks, --spec-type mtp plumbing) the canonical place is the upstream [PR #1744 thread](https://github.com/ikawrakow/ik_llama.cpp/pull/1744) or the [ikawrakow/ikllama.cpp` issue tracker](https://github.com/ikawrakow/ikllama.cpp/issues). For upstream weights / chat-template / tokenizer questions, file against google/gemma-4-26B-A4B-it-assistant — but please filter quant-format problems out before going there; Google does not maintain the GGUF tooling.