urs-rs/gemma-4-26B-A4B-it-FP8-DYNAMIC

Gemma 4 26B-A4B-it FP8

A production-grade FP8 dynamic quantization of google/gemma-4-26B-A4B-it using llm-compressor's oneshot() method with the load_context() MoE linearization pattern.

Model Details

• —Base Model: google/gemma-4-26B-A4B-it
• —Quantization: FP8 Dynamic (W8A8 per-token)
• —Method: llm-compressor oneshot() with load_context()
• —Quantized Layers: All Linear layers except router projections, norms, and lm_head
• —Router Weights: Kept in full BF16 precision for numerical stability in expert dispatch
• —Target Runtime: vLLM 0.20.2rc1+ with cu129 (CUDA 12.9)

Quantization Details

This checkpoint was quantized using llm-compressor's FP8 dynamic quantization. The quantization process carefully preserves model quality by keeping router weights and normalization layers in full precision to maintain numerical stability.

Quantization Specs:

• —Format: float-quantized (FP8)
• —Weights: 8-bit, channel-wise, symmetric, per-tensor static
• —Activations: 8-bit, token-wise, symmetric, dynamic
• —Preserved Layers: Router projections, normalization layers, and language model head (full BF16)
• —Compressed Size: ~13.3 GiB (47% reduction vs BF16)

Usage with vLLM

Basic Launch

docker run -d --name gemma4-fp8 \
  --ipc=host --network host --shm-size 16G --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:gemma4-0505-cu129 \
  --model urs_rs/gemma-4-26B-A4B-it-FP8-DYNAMIC \
  --kv-cache-dtype auto \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.95 \
  --host 0.0.0.0 --port 8000

Benchmarks

VRAM Usage & Concurrency (H100 NVL, TP=1)

• —Model Size: ~13.3 GiB (47% reduction vs BF16)
• —KV Cache Budget: ~54 GiB available
• —Max Concurrency (16K context): 21.3x
• —Throughput Gain vs BF16: ~80% more concurrent calls at equivalent latency

Accuracy (vs BF16)

• —MMLU-Pro: -0.26 percentage points (negligible)
• —Generation Quality: No observable drift on multi-turn conversations
• —Router Stability: Keeping router weights in BF16 eliminates topic drift in long-context calls

Key Design Decisions

Why load_context() + oneshot()?

The raw google/gemma-4-26B-A4B-it checkpoint stores MoE experts as fused 3D tensors [128, dim, dim] rather than separate nn.Linear modules. Generic quantization tools (including model_free_ptq) cannot handle this format: they linearize for calibration but fail to fuse weights back to the native format on save, leaving the checkpoint in a broken intermediate state.

Using load_context() with oneshot() triggers llm-compressor's internal linearize_moe() and automatic fuse-back-on-save path, which correctly handles Gemma 4's fused expert structure.

Why exclude router weights?

Router weights make discrete expert-selection decisions. Quantizing them to FP8 adds noise to this decision boundary, which compounds over long sequences (especially in voice call contexts with many tokens). Keeping them in full BF16 ensures numerically stable expert dispatch regardless of sequence length.

Hardware Requirements

• —Minimum: NVIDIA A100 (40GB) or H100 (80GB)
• —Tested: NVIDIA H100 NVL with vLLM 0.20.2rc1+
• —CUDA Driver: 12.7+ (cu129 image requires 12.9 compatible driver)

Citation

@article{gemma4,
  title={Gemma 4: Open Models for Everyone},
  author={Google DeepMind},
  year={2025}
}

@software{llm-compressor,
  title={LLM Compressor: Compression of Large Language Models},
  author={Neural Magic},
  url={https://github.com/vllm-project/llm-compressor}
}

License

This quantized checkpoint is provided under the same license as the base google/gemma-4-26B-A4B-it model (Gemma Community License). See https://huggingface.co/google/gemma-4-26B-A4B-it for full license terms.

Questions & Support

For issues specific to this quantization:

1. 1.Check the llm-compressor docs
2. 2.File an issue on the vLLM GitHub
3. 3.See the Greylabs repo discussions

Last Updated: July 2026 Quantized by: u.rajsingh2503@gmail.com