Modelpublic

TheStageAI/gemma-4-E4B-it-qat-GGUF

sourceHugging Facemitupdated 4mo agoView on Hugging Face

1likes180downloads

Model Card

TheStageAI/gemma-4-E4B-it-qat-GGUF

A portable GGUF release of Google's Gemma 4 E4B instruction model, compressed from Google's QAT-trained BF16 weights and emitted as standard llama.cpp-compatible .gguf files.

—Run it with: llama.cpp or other GGUF-compatible runtimes
—Compression source: `google/gemma-4-E4B-it-qat-q4_0-unquantized`
—BF16 reference: `google/gemma-4-E4B-it`
—Smaller native release: `TheStageAI/gemma-4-E4B-it-qat`

Use this repo when deployment portability matters most. If you can run our native MLX runtime and want the smallest artifacts, use the edge-lm sibling release.

Why this exists

The native edge-lm checkpoints use custom codecs for both decoder weights and PLE tables, which is why they are smaller at comparable quality. Many deployments, however, need standard GGUF files that work with llama.cpp-compatible tooling.

This repo keeps the production bit-width schedules from our native compression pipeline, but maps the weights into GGUF-compatible quantization formats. The result is larger than the native release, but portable.

How it was compressed

We start from Google's QAT-trained BF16 checkpoint and reuse the production m and l schedules from the native release.

—Transformer blocks - the M and L files follow our RCO-selected production bit-width schedules, then emit the weights in GGUF-compatible K-quant layouts with the required group sizes and symmetric/asymmetric modes for each tensor family.
—PLE tables - stored with GGUF-compatible Q4 scalar quantization instead of the native AQLM PLE codec, so the files stay portable across GGUF runtimes.
—Token embeddings / LM head - quantized through the same GGUF-compatible path as the rest of the model.
—W4-uniform - a conservative uniform 4-bit GGUF variant with the same Q4 PLE path.

Operating points

File	Trade-off	Size	Compression vs BF16	Transformer	PLE
`gemma-4-E4B-it-qat-GGUF-M.gguf`	Compact GGUF target	4.08 GB	3.9x	production `m` mapped to GGUF	GGUF Q4
`gemma-4-E4B-it-qat-GGUF-L.gguf`	Higher-quality GGUF target	4.40 GB	3.6x	production `l` mapped to GGUF	GGUF Q4
`gemma-4-E4B-it-qat-GGUF-W4-uniform.gguf`	Uniform W4 baseline	4.40 GB	3.6x	uniform W4 GGUF	GGUF Q4

Usage

Use a recent upstream llama.cpp build. Example:

bash

llama-completion \
  -m gemma-4-E4B-it-qat-GGUF-L.gguf \
  -p "Explain gravity in one sentence." \
  -n 64

Benchmarks

For quality evaluation, GGUF checkpoints are converted through the same dequantized BF16 evaluation path used for the native release, so the backend is equalized. IFEval p/i means prompt strict / instruction strict, using the corrected public recipe with max_gen_toks=1280.

Model	Size	Compression	MMLU-Pro	IFEval p/i
BF16 reference	15.88 GB	1.0x	70.49	85.03 / 89.57
GGUF `M`	4.08 GB	3.9x	65.44	81.33 / 87.29
GGUF `L`	4.40 GB	3.6x	67.90	82.81 / 87.89
GGUF `W4-uniform`	4.40 GB	3.6x	68.34	81.33 / 87.05

MMLU-Pro is the official checkpoint-wise vLLM route with Gemma chat formatting and thinking enabled. The .gguf files in this repo also passed generation smoke tests with upstream llama.cpp.

Files

File	Contents
`gemma-4-E4B-it-qat-GGUF-M.gguf`	Compact GGUF target
`gemma-4-E4B-it-qat-GGUF-L.gguf`	Higher-quality GGUF target
`gemma-4-E4B-it-qat-GGUF-W4-uniform.gguf`	Uniform W4 GGUF baseline

License

Released under the MIT License. As a derivative of Gemma, the weights are also subject to the Gemma Terms of Use.

Citation

If you use these checkpoints, please cite the Gemma 4 release and the methods we build on (GPTQ, QEP, AQLM, RCO) - see the references in the `edge-lm` write-up.