dongchengli/gemma-3-4b-it-text-int4-gptq

gemma-3-4b-it-text-int4-gptq

A text-only, GPTQ INT4 (W4A16) quantization of google/gemma-3-4b-it, built and verified to run on vLLM 0.26.0 / CUDA 13.

How this model was built

Two deliberate steps, in this order:

google/gemma-3-4b-it (bf16, multimodal, 8.1 GB)
        │
        ├─ 1. STRIP VISION  ──  vision_tower + multi_modal_projector removed;
        │                       language model extracted into a standalone
        │                       Gemma3ForCausalLM checkpoint
        ▼
   text-only bf16
        │
        ├─ 2. QUANTIZE  ─────  GPTQ W4A16, group size 128, symmetric,
        │                      512 calibration samples, lm_head left in bf16
        ▼
  text-only INT4  (2.8 GB)

Step 1 is a real removal, not a runtime flag. The vision components are gone from the checkpoint. That is why this is a Gemma3ForCausalLM and not a Gemma3ForConditionalGeneration: smaller download, smaller resident footprint, and no multimodal code paths at load time. It also sidesteps a practical problem — the SigLIP vision encoder has layers whose dimensions are not divisible by the quantization group size, so quantizers either error out or have to special-case them.

Step 2 then quantizes only what remains, so the whole 4-bit budget is spent on the language model.

Note on the base model. This is built from the standard gemma-3-4b-it bf16 weights (obtained via the unsloth mirror). It is not derived from Google's QAT (quantization-aware training) release. If you need QAT-grade INT4 quality, start from google/gemma-3-4b-it-qat-int4-unquantized instead.

Why this exists

At the time of upload, quantized gemma-3-4b-it checkpoints were awkward to serve on current vLLM:

• —Older vLLM (0.8.x), which is what you are stuck with on a CUDA 12.6 driver, produced only `<pad>` tokens for Gemma 3 in our testing.
• —An existing community AWQ build of the same base model produced degenerate output ("Paris, it is the the is is the is is is...") on vLLM 0.26.0 in our testing, i.e. the AWQ + Gemma 3 path was not usable for us.

This checkpoint uses compressed-tensors / GPTQ instead and produced coherent output in every check we ran. Your mileage on other engines may differ — see Scope of testing below.

Usage

vllm serve <this-repo> \
    --served-model-name gemma-3-4b-it \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.90

Requires a driver new enough for the CUDA build of your vLLM. On a CUDA 13 wheel that means driver >= 580. On a 560 driver you will be pushed back to vLLM 0.8.x, where we could not get Gemma 3 to generate valid tokens at all.

from vllm import LLM, SamplingParams

llm = LLM(model="<this-repo>", max_model_len=4096)
out = llm.chat(
    [{"role": "user", "content": "Explain gravity in two sentences."}],
    SamplingParams(max_tokens=256, temperature=0.3),
)
print(out[0].outputs[0].text)

Throughput

Measured on a single RTX A4000 (16 GB, Ampere, SM 86), vLLM 0.26.0, --max-model-len 4096, --gpu-memory-utilization 0.90, chat endpoint, short prompts, 128 output tokens per request, measured client-side on localhost:

Reproduced on a second, identically specced machine: ~3736 tok/s at 3072 concurrency.

These are aggregate numbers under heavy batching. Per-request latency at 3072 concurrency is tens of seconds; for interactive use keep concurrency in the low hundreds, where aggregate throughput is already >2000 tok/s and per-request latency stays in the seconds range.

Throughput is workload- and hardware-dependent. Do not read these as a claim about any other model, engine, quantization, or GPU — we did not run a controlled comparison against other quantization schemes.

Scope of testing — please read

What we did verify:

• —3072 concurrent generations: 0 empty completions, 0 degenerate repetition loops out of 3072, by automated check plus manual sampling.
• —Coherent, correct output in English and Chinese, including code generation and a 3018-token input summarized correctly.
• —Stable serving across restarts on two machines.

What we did not do:

• —No perplexity measurement. No wikitext or equivalent comparison against the bf16 parent.
• —No benchmark evaluation. No MMLU, HellaSwag, GSM8K, or any lm-evaluation-harness task.
• —No long-context stress beyond ~3k tokens.
• —No evaluation on languages other than English and Chinese.

Therefore: this card makes no accuracy claim. GPTQ W4A16 is lossy by construction. If accuracy matters for your use case, evaluate it on your own task before deploying. If you do run evaluations, please open a discussion — the numbers would improve this card.

How it was produced

1. 1.Loaded gemma-3-4b-it in bf16 and extracted the language model into a standalone Gemma3ForCausalLM, dropping vision_tower and multi_modal_projector.
2. 2.Quantized with llm-compressor 0.12.0.1:

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier

recipe = GPTQModifier(targets="Linear", scheme="W4A16", ignore=["lm_head"])
oneshot(
    model=model, processor=tokenizer, dataset=ds,
    recipe=recipe, max_seq_length=2048, num_calibration_samples=512,
)

Calibration: 512 samples from HuggingFaceH4/ultrachat_200k (train_sft), chat template applied, max sequence length 2048. Group size 128, symmetric, static activation ordering, dampening fraction 0.01.

License

This is a derivative of Google's Gemma 3 and remains governed by the [Gemma Terms of Use](https://ai.google.dev/gemma/terms), including the Prohibited Use Policy. By using this model you agree to those terms, which carry over to any further redistribution. This is a modified version: the vision components have been removed and the weights have been quantized to INT4.