srmiles/gemma-4-12B-it-assistant-GGUF

Gemma 4 12B-it Assistant (MTP Drafter) — GGUF (BF16)

Correctly-converted BF16 GGUF of Google's official `google/gemma-4-12B-it-assistant` MTP (Multi-Token Prediction) drafter, for use with llama.cpp speculative decoding.

Pair this drafter with the Gemma 4 12B target model for a substantial decode-throughput uplift with mathematically identical output quality.

Why this GGUF exists

The community GGUFs for Google's Gemma 4 assistants use an architecture string mismatch — gemma4_assistant (underscore) instead of upstream llama.cpp's gemma4-assistant (hyphen) — which makes them fail to load on any modern llama.cpp build.

This GGUF was converted directly from Google's official BF16 safetensors with llama.cpp's own `convert_hf_to_gguf.py` (b10215 / commit eb41d503b), so every metadata key is correctly namespaced and it loads cleanly on upstream llama.cpp.

Verified working

• —llama.cpp SYCL b10215+ on Intel Arc Pro B60 (Battlemage / Xe2, 24 GB)
• —Should work on any llama.cpp backend at b10215 or newer.

Usage (llama.cpp)

llama-server \
  -m gemma-4-12B_q4_0-it.gguf \
  --model-draft gemma-4-12B-it-assistant-official.bf16.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  -ngl 99 -c 8192 -fa on -ub 2048 -b 2048 \
  --jinja --reasoning off \
  --host 0.0.0.0 --port 8000

Benchmark (Intel Arc Pro B60, llama.cpp SYCL b10215)

Measured on real workload (2-5K token prompts, structured JSON output):

Compared to 12B without a drafter: +257% decode, +530% prefill on b10215.

Full bench methodology + comparison to Ornith 9B production model: github.com/srmiles/local-llm-benchmarks.

Conversion recipe (reproducible)

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && git checkout b10215
pip install --index-url https://download.pytorch.org/whl/cpu torch

hf download google/gemma-4-12B-it-assistant --local-dir gemma-4-12B-it-assistant-hf
python convert_hf_to_gguf.py gemma-4-12B-it-assistant-hf/ \
  --outfile gemma-4-12B-it-assistant-official.bf16.gguf \
  --outtype bf16

Files

• —gemma-4-12B-it-assistant-official.bf16.gguf — 862 MB, BF16

Credits

• —Model weights: Google — `google/gemma-4-12B-it-assistant`
• —MTP research: DeepMind — Gemma 4 MTP documentation
• —Conversion tool: ggml-org/llama.cpp convert_hf_to_gguf.py at commit eb41d503b (b10215)
• —Conversion & bench: srmiles/local-llm-benchmarks

License

Apache 2.0.