ji-farthing/gemma-4-qat-q4_0-MTP-assistants-ik-llama-GGUF

Gemma 4 QAT Q40 MTP Assistants for ikllama

This repo contains ik_llama-compatible GGUF assistant/draft models converted from Google's Gemma 4 QAT Q4_0 assistant checkpoints.

These are not standalone chat models. Use them as --model-draft files next to the matching Google Gemma 4 QAT Q4_0 target GGUF.

Do not load this repository as the primary model. Hugging Face may show generic GGUF usage snippets for this repo, but these files are assistant/draft GGUFs only.

Important E2B/E4B Caveat

The E2B and E4B files are experimental edge-model assistants. They converted and smoke-tested successfully with matched QAT targets after an ik_llama shared-KV target loader fix.

Current ik_llama main includes that loader fix via PR #1927. Older builds may fail before draft loading with errors such as:

E2B: missing blk.15.attn_k.weight
E4B: missing blk.24.attn_k.weight

Those target tensors appear intentionally absent in Google's QAT edge GGUFs: the E2B/E4B targets advertise shared-KV tail layers.

E4B has an additional runtime caveat. On current ik_llama main, local tests could load E4B QAT + matched E4B assistant and generate in a small CPU/offload smoke path. The longer full-GPU flash-attention path was still unstable before acceptance counters in our RTX 4070 test. Treat E4B as experimental until the remaining runtime/FA path is resolved upstream.

Use matched pairs only. The E2B assistant is not expected to work with the E4B target, and the E4B assistant is not expected to work with the E2B target.

Files

BF16 conversion intermediates and Q80 reference variants are intentionally not published here. The Q40 files are the intended pairing for Google's QAT Q4_0 target GGUFs.

Matching Target Repos

Use the assistants with Google's official target GGUFs:

• —google/gemma-4-E2B-it-qat-q4_0-gguf
• —google/gemma-4-E4B-it-qat-q4_0-gguf
• —google/gemma-4-12B-it-qat-q4_0-gguf
• —google/gemma-4-26B-A4B-it-qat-q4_0-gguf
• —google/gemma-4-31B-it-qat-q4_0-gguf

Example

llama-server \
  -m /path/to/gemma-4-12b-it-qat-q4_0.gguf \
  --model-draft /path/to/gemma-4-12B-it-qat-q4_0-MTP-ik_llama-Q4_0.gguf \
  --spec-type mtp:n_max=4,p_min=0.0 \
  --jinja

Use a current ik_llama build with Gemma 4 MTP support. Reproducing these conversions from Google's safetensors currently requires convert_hf_to_gguf.py to recognize Gemma4UnifiedAssistantForCausalLM as the existing Gemma 4 MTP assistant converter path.

Validation

Local conversion metadata checks:

All published files report general.architecture=gemma4_mtp.

Historical runtime smoke on an RTX 4070 with ik_llama build 4561 (6b9de3dba):

Additional current-main check with ik_llama bbe1a511e:

Conversion Notes

Source assistant repos:

• —google/gemma-4-E2B-it-qat-q4_0-unquantized-assistant
• —google/gemma-4-E4B-it-qat-q4_0-unquantized-assistant
• —google/gemma-4-12B-it-qat-q4_0-unquantized-assistant
• —google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant
• —google/gemma-4-31B-it-qat-q4_0-unquantized-assistant

The assistants were converted through ik_llama's Gemma 4 MTP assistant converter, then quantized with llama-quantize to Q4_0.