FreedomAISVR/Gemma-4-E2B-it-QAT-NVFP4-GGUF

Gemma 4 E2B Instruct - QAT + NVFP4 Hybrid GGUF

QAT-optimized weights preserved at Q4_0, overhead tensors quantized to NVFP4.

What Makes This Different

This is a hybrid quantization of Google official QAT (Quantization-Aware Training) model. Instead of requantizing the Q4_0 weights (which breaks QAT benefits and vision quality), we:

1. 1.Kept all weight tensors at Q4_0 - attention, FFN, embeddings - exactly as Google trained them
2. 2.Quantized only the F32 norm/bias tensors to NVFP4 - these are the overhead tensors (layer norms, RMS norms, etc.)
3. 3.Used Google QAT mmproj - the vision projector trained alongside the QAT model

Why Standard NVFP4 from QAT Breaks Vision

Google QAT model was specifically trained to be resilient to Q40 quantization patterns. The weight values learned during QAT compensate for Q40 rounding. When you requantize Q4_0 -> F32 -> NVFP4, a second round of quantization error is introduced that QAT training did not account for. Vision tokens flow through the same attention/FFN layers - precision loss disproportionately degrades vision.

How the Hybrid Approach Works

Using llama-quantize --tensor-type-file with --allow-requantize:

llama-quantize --allow-requantize --tensor-type-file keep_q4.txt input.gguf output.gguf NVFP4

The tensor-type-file lists all Q40/Q4K tensors to keep at their current type. When the quantizer sees curtype == newtype, it copies the tensor data as-is - zero precision loss. Only the remaining F32 tensors are quantized to NVFP4.

Usage

# llama.cpp
llama-server -m gemma-4-E2B-it-qat-nvfp4.gguf --mmproj mmproj-gemma-4-E2B-it-qat.gguf -ngl 99

Source

• —Base model: google/gemma-4-E2B-it-qat-q4_0-unquantized
• —Quantized with: llama.cpp build 537 (commit d2c6795)
• —Chat template: Native Gemma 4 (thinking enabled by default)
• —Vision: Full multimodal support via QAT mmproj

Files

License

Apache 2.0 (same as base model)