Danny-Dasilva/gemma-4-12B-it-W4A16-GPTQ-g32-DSpark

gemma-4-12B-it — W4A16 GPTQ (group=32) · DSpark-verified

A post-training GPTQ int4 (W4A16, group_size=32, symmetric) quantization of `google/gemma-4-12B-it`, built specifically to be a [DSpark](https://github.com/deepseek-ai/DeepSpec) speculative-decoding target (draft head: `deepseek-ai/dspark_gemma4_12b_block7`).

TL;DR: 198 tok/s single-stream on one RTX 5090 (32 GB) — the fastest gemma-4-12B configuration we measured on that card, at 7.8 GB of weights.

Why this exists — QAT breaks DSpark, post-training quant doesn't

The DSpark draft head was trained against the bf16 gemma-4-12B-it weights. Any retraining of the target — including the official QAT checkpoint `google/gemma-4-12B-it-qat-w4a16-ct` — drifts far enough that draft acceptance collapses and speculation makes the model slower:

Post-training rounding preserves draft alignment; QAT retraining destroys it. If you pair gemma-4-12B int4 with the DeepSeek DSpark draft, you need a post-training quant like this one.

Measured speed — RTX 5090 (32 GB, Blackwell sm_120), single stream (bs=1)

vLLM (recent main with gemma-4 DSpark support), FlashInfer target attention, greedy, 256-token generations over 4 chat prompts, CUDA graphs (piecewise):

Weights: 7.8 GB (vs 24 GB bf16, 13 GB fp8) — target + draft + KV fit comfortably in 32 GB with CUDA graphs, which bf16 cannot do.

Usage (vLLM + DSpark)

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer

MODEL = "Danny-Dasilva/gemma-4-12B-it-W4A16-GPTQ-g32-DSpark"
DRAFT = "deepseek-ai/dspark_gemma4_12b_block7"

llm = LLM(
    model=MODEL,
    max_model_len=8192,
    attention_backend="FLASHINFER",       # gemma-4 full-attn layers have head_dim 512
    enable_flashinfer_autotune=True,      # +2% in our runs
    speculative_config={
        "method": "dspark",
        "model": DRAFT,
        "num_speculative_tokens": 7,      # = draft block size; k<7 or k=14 are both slower
        "attention_backend": "TRITON_ATTN",
    },
)

tok = AutoTokenizer.from_pretrained(MODEL)
prompt = tok.apply_chat_template(
    [{"role": "user", "content": "Explain transformers step by step."}],
    add_generation_prompt=True, tokenize=False,
)
print(llm.generate([prompt], SamplingParams(temperature=0.0, max_tokens=256))[0].outputs[0].text)

Notes:

• —Always apply the chat template — gemma-4-it uses <|turn>-style tokens and add_bos_token=False; raw prompts produce garbage on any backend.
• —num_speculative_tokens must be ≤ 7 or a multiple of 7 (draft block size). 7 is optimal.
• —Works without DSpark too, as a normal compressed-tensors W4A16 checkpoint (kernel path identical to the official QAT release: group=32, symmetric, pack-quantized).
• —kv_cache_dtype="fp8" measured slower (−3%) at short context on this card.

Quantization recipe

llm-compressor 0.12 GPTQ, one-shot:

• —256 calibration samples from HuggingFaceH4/ultrachat_200k (trainsft), chat template applied, maxseq_length 2048
• —scheme: int4, group_size=32, symmetric, weights-only (W4A16), pack-quantized format — mirrors the official QAT checkpoint's config so every vLLM kernel path is identical
• —sequential_targets=["Gemma4UnifiedTextDecoderLayer"]; lm_head and vision/audio embedder projections kept in bf16 (same 17-entry ignore list as the official QAT release, spelled so both HF and vLLM module names match)
• —processor_config.json included (vLLM's multimodal processor init requires it)

Provenance

• —Base: google/gemma-4-12B-it @ 5926caa4 (bf16, untouched — no finetuning)
• —Quantized 2026-07-04 on a single RTX 5090; outputs verified coherent against bf16 generations
• —Gemma is provided under and subject to the Gemma Terms of Use: https://ai.google.dev/gemma/terms