Vishva007/gemma-4-12B-it-W4A16-AutoRound-GPTQ

Gemma 4 12B IT - W4A16 AutoRound Quantized Variants

This repository hosts 4-bit weight, 16-bit activation (W4A16) quantized variants of google/gemma-4-12B-it. The models were quantized using Intel's AutoRound framework tailored specifically for the architectural requirements of the Gemma 4 family.

Available formats in the series:

• —Vishva007/gemma-4-12B-it-W4A16-AutoRound (Standard AutoRound format)
• —Vishva007/gemma-4-12B-it-W4A16-AutoRound-AWQ (AWQ format conversion)
• —Vishva007/gemma-4-12B-it-W4A16-AutoRound-GPTQ (GPTQ format conversion)

Architectural Advantage: Gemma 4 12B Unified

The Gemma 4 12B Unified model features a ground-breaking encoder-free multimodal architecture. Unlike traditional vision-language models that rely on separate heavy visual/audio encoders (like ViT or Whisper), the 12B Unified model projects raw image patches and audio waveforms directly into the main LLM's embedding space via lightweight linear layers.

Because text, image, and audio flow natively into a single decoder-only transformer, this model benefits dramatically from weight-only quantization, offering minimal multimodal latency and a highly streamlined memory footprint.

Quantization Recipe & Environment

The models were generated using the following computational setup and tuning parameters. Notably, it leverages RTN mode (`iters: 0`) with non-text module protection, which is mandatory for maintaining performance stability on the Gemma 4 native multimodal layers.

Environment Setup

• —PyTorch Version: 2.10.0+cu128
• —CUDA Version: 12.8 (CUDA Available: True)
• —Hardware: NVIDIA L40 (48 GB VRAM)

Tuning Configuration

TUNING_CONFIG = {
    "group_size": 128,
    "sym": True,
    "iters": 0,                      # RTN mode — required for Gemma 4 stability
    "disable_opt_rtn": True,
    "nsamples": 256,
    "seqlen": 2048,
    "low_gpu_mem_usage": False,
    "quant_nontext_module": False,   # Keeps linear projection layers intact for vision/audio inputs
    "layer_config": layer_config,
}

Deployment & Inference Guide

Due to the unique unified nature of Gemma 4's multimodal inputs, deployment requires the specialized vLLM unified image containing updated parser configurations.

Running with Docker (vLLM)

Deploy a production-ready OpenAI-compatible API server using the custom vllm-openai:gemma4-unified engine:

docker run --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:gemma4-unified \
    --model Vishva007/gemma-4-12B-it-W4A16-AutoRound \
    --quantization autoround \
    --port 8000 \
    --max-model-len 4096 \
    --trust-remote-code

(Note: Change `--quantization` flag to `awq` or `gptq` depending on the specific variant file path you mount).

🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.

PyTorch 2.14

PyTorch 2.13

PyTorch 2.12

🛠️ Reproduction & Source Code

The complete script used to perform this quantization, manage the layer configurations, and optimize the memory footprint is fully open-sourced.

You can find the step-by-step Jupyter Notebook containing the exact execution environment, dependency setups, and layer configurations in the GitHub repository:

👉 [AutoRound-Quantization: Gemma 4 12B RTN Notebook](https://github.com/vishvaRam/AutoRound-Quantaization/blob/main/Gemma4/auto_round_Gemma4-12B-RTN.ipynb)

How to Reproduce

To replicate this quantization run on your own hardware (e.g., NVIDIA L40 or similar 40GB+ VRAM instances), clone the setup and run the notebook:

git clone [https://github.com/vishvaRam/AutoRound-Quantaization.git](https://github.com/vishvaRam/AutoRound-Quantaization.git)
cd AutoRound-Quantaization/Gemma4