GotoAI-Inc/gemma-4-31B-it-W8A16

gemma-4-31B-it-W8A16

Int8 weight-only quantization of google/gemma-4-31B-it, in compressed-tensors format for vLLM. 33.72 GB, down from 62.55 GB — it fits a 48 GB card with room for a 128k context window.

This is the fidelity-first build. Int8 round-to-nearest stays far closer to the bfloat16 weights than int4 does, at ~1.8x the footprint of the int4 W4A16 sibling (19.07 GB). If you are targeting a 24 GB card, use that one — or Google's QAT w4a16 build (23.27 GB), which is quantization-aware trained and is the better 4-bit checkpoint where it fits. Use this one when you have the VRAM and want the least quality loss quantization can give without calibration.

Unofficial and unaffiliated with Google. All model capabilities, evaluations and limitations belong to the original model card — see the base model for those.

What was changed

Weights were quantized from bfloat16 to int8, group size 128, symmetric, weight-only (activations stay 16-bit) using llmcompressor.model_free_ptq. No calibration data was used and the model was never loaded — the quantizer operates directly on the safetensors. Architecture, tokenizer, chat template and processor config are the vendor's, unmodified.

410 Linear modules were converted — 93.6% of the source checkpoint's bytes, landing as 88.2% of this one's:

Of the quantized share, 29.29 GB is packed int8 and 0.46 GB is bf16 group scales — the same scale count as the W4A16 sibling, since both use group size 128.

The 410 modules are 7 per layer (q/k/v/o_proj, gate/up/down_proj) on the 50 sliding layers and 6 on the 10 global ones, which have no v_proj at all (see Fitting the card).

Left at bfloat16:

• —`vision_tower`, `embed_vision` — vLLM's Gemma 4 loader builds the towers unquantized, so a checkpoint carrying quantized vision weights is asking for a version-dependent load failure. The tower's intermediate_size of 4304 is not divisible by the group size of 128 in any case.
• —`embed_tokens` — precision-sensitive, and it is the output head here (tie_word_embeddings: true, and no lm_head tensor exists in the checkpoint).

Unlike the 12B, this model has no audio tower (audio_config: null, zero audio tensors — Gemma 4 ships audio only on E2B, E4B and 12B), so the profile's re:.*audio.* pattern matches nothing here. It also uses a conventional dedicated vision encoder rather than the 12B's encoder-free "Unified" design, which is why the tower shows up as 1.15 GB of separate weights.

Usage

vllm serve GotoAI-Inc/gemma-4-31B-it-W8A16 \
  --max-model-len 65536 \
  --enable-auto-tool-choice --tool-call-parser gemma4 \
  --reasoning-parser gemma4

Do not pass --quantization; compressed-tensors is detected from config.json. The int8 W8A16 scheme uses Marlin kernels and runs on compute capability 7.5 and above.

Version note. Gemma 4 needs transformers >= 5.10.1, but vLLM ≤ 0.27.1 cannot serve Gemma 4 with transformers >= 5.15: at 5.15 the config turns global_head_dim into a per-layer head_dim override, and older vLLM reads head_dim globally, raising AmbiguousGlobalPerLayerAttributeError at engine-config time. Use either transformers < 5.15 with vLLM 0.25.1–0.27, or vLLM >= 0.28, which handles both layouts. The architecture itself (Gemma4ForConditionalGeneration) and both gemma4 parsers are present from 0.25.1 on.

Fitting the card

Gemma 4 interleaves 50 sliding-attention layers (window 1024, 16 KV heads, head_dim 256) with 10 global layers (every 6th, 4 KV heads, global_head_dim 512). The sliding layers' cache is bounded by the window at roughly 0.8 GB per sequence regardless of context length; only the global layers grow, at about 80 KB/token:

A 48 GB card is comfortable to 128k; 256k needs 80 GB. A 40 GB card does not have the headroom for the full 33.72 GB of weights plus a useful cache — add --language-model-only (which frees the 1.15 GB tower, since vLLM skips tower weights entirely when every modality limit is zero, plus the multimodal profiling headroom) and keep the context near 32k, or use the W4A16 sibling instead. This is arithmetic from config.json, not a measured deployment.

Note that the global layers use unified keys and values (attention_k_eq_v: true, and the checkpoint has no v_proj on those layers). That saves weight bytes, but vLLM loads the K weights into both the K and V slots, so the cache still holds both copies — the table above already assumes that.

Thinking

The chat template defaults enable_thinking to false, so this model does not think unless asked. Both knobs are template variables passed through chat_template_kwargs:

{"chat_template_kwargs": {"enable_thinking": true}}      // injects <|think|> into the system turn
{"chat_template_kwargs": {"preserve_thinking": true}}    // keep thinking on tool-call turns

With thinking on, reasoning is emitted as <|channel>thought … <channel|>, which --reasoning-parser gemma4 splits into reasoning_content. Per the base model card, thinking from earlier turns should not be replayed into history — except on tool-call turns, which is exactly what preserve_thinking keeps.

Sampling, per the base model card: temperature=1.0, top_p=0.95, top_k=64 for all use cases. Place image content before the text in a prompt. The visual token budget is configurable (70/140/280/560/1120, default 280) — lower it for video and captioning, raise it for OCR and document parsing.

Reproducing this checkpoint

Built with llm-quantizer:

./llmq.py run --profile gemma-4-31b-it --scheme W8A16

which re-shards the source — it ships as 2 shards, the larger 49.78 GB, which no consumer GPU can hold — into 17 pieces of ~4 GB, then:

# llmcompressor==0.13.1a20260814, compressed-tensors==0.18.1a20260818,
# transformers==5.15.1, torch==2.13.0
from llmcompressor import model_free_ptq

model_free_ptq(
    model_stub="gemma-4-31B-it-resharded",
    save_directory="gemma-4-31B-it-W8A16",
    scheme="W8A16",
    ignore=["re:.*vision.*", "re:.*audio.*", "lm_head", "re:.*embed_tokens.*"],
    device="cuda:0",
)

A job holds one shard at a time, so after re-sharding the build peaks at a few GB of VRAM rather than the model size — no large GPU required.

Evaluation

No benchmarks have been run. Data-free round-to-nearest quantization degrades quality more than a calibrated (GPTQ/AWQ) or QAT build; how much, for your task, is unmeasured here. Int8 degrades far less than int4 — that is the reason this build exists — but "less" is not "none". Treat published Gemma 4 benchmark numbers as describing the bf16 model, not this one.

For an agentic model the informative checks are well-formed reasoning_content and clean multi-step tool calls rather than perplexity: structured emission degrades before fluency does.

License

Apache 2.0, inherited from the base model — see LICENSE and Google's Gemma 4 license page. The base repository ships no LICENSE file, so the Apache-2.0 text is included here for redistribution. "Gemma" is Google's mark; this repository is not endorsed by or affiliated with Google.