necroyancer/gemma-4-31B-it-NVFP4-turbo-vision

gemma-4-31B-it-NVFP4-turbo-vision

A more compact NVFP4 quantization of google/gemma-4-31B-it that keeps the vision tower intact so multimodal (image + text) input still works.

This is a vision-preserving variant of LilaRest's excellent `gemma-4-31B-it-NVFP4-turbo`. Same RTN-on-attention recipe; the only difference is that the BF16 vision tower, vision embedding projection, and vision_config are retained, and the architecture stays Gemma4ForConditionalGeneration. For the recipe rationale, hardware requirements, and architecture compatibility matrix, see LilaRest's model card — it's the canonical reference.

What's quantized vs. preserved

The RTN step is byte-deterministic from the source weights — no calibration dataset, no forward passes, no training dynamics. Pure tensor math.

Recipe details (RTN factorization)

For each BF16 attention weight tensor:

• —weight_scale_2 = amax(weight) / 6 (per-tensor FP32 scalar)
• —weight_scale = block_amax / (6 · weight_scale_2) (per-block FP8 e4m3, block size 16) — values land in [0, 1] without saturating FP8
• —input_scale = 1.0 (placeholder; signals "no calibration" to the modelopt loader)
• —Weight packed as uint8 FP4 e2m1 with two values per byte

This matches the convention LilaRest used for his -turbo quant. The default modelopt formula weight_scale_2 = amax / (6·448) is tuned for calibrated PTQ (where activation absorption inflates per-block amax) and saturates FP8 for low-amax weights like attention projections.

Reconstruction quality vs. source BF16: mean abs error 0.00114, max 0.0259 — fractionally better than the published -turbo quant (0.00117 / 0.0283).

Differences from LilaRest's original

Everything else is identical: tokenizer, chat template, generation config, quantization_config schema, kv-cache scheme, sharding, modelopt producer version (0.37.0).

Serving with vLLM

Tested on vllm/vllm-openai:gemma4 (vLLM 0.19.1.dev6, Torch 2.10.0+cu129, Transformers 5.5.0). The gemma4-cu130 image works equally well.

Minimal command (adapt the runtime flags to your workload):

vllm serve necroyancer/gemma-4-31B-it-NVFP4-turbo-vision \
  --quantization=modelopt \
  --max-model-len=65536 \
  --kv-cache-dtype=fp8 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --trust-remote-code \
  --enable-auto-tool-choice \
  --tool-call-parser=gemma4 \
  --reasoning-parser=gemma4

Vision input works through the standard OpenAI-compatible chat completions API (content parts with image_url).

Benchmark (RTX PRO 6000, vLLM vllm bench throughput)

Apples-to-apples vs LilaRest's published numbers:

Within measurement noise (+1.9%). The retained vision tower adds ~1.5 GiB to peak VRAM but doesn't impact text-only throughput — it's loaded but bypassed on text prompts.

Additional reference points on this hardware:

Credits

• —LilaRest for the `gemma-4-31B-it-NVFP4-turbo` recipe — the RTN factorization, the empirical validation that attention RTN is essentially free, and the weight_scale_2 = amax/6 choice that avoids FP8 saturation. This variant is a direct application of his work.
• —NVIDIA for `Gemma-4-31B-IT-NVFP4`, the calibrated MLP-quant base.
• —Google for `gemma-4-31B-it`.

License

Inherits the Gemma license from the upstream google/gemma-4-31B-it model. Use accordingly.