CoolFace
Modelpublic

sphaela/gemma-4-E2B-it-AutoRound-GGUF

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face
0likes292downloads
Model Card

Gemma-4-E2B-it GGUF (AutoRound Quantized)

This repository contains GGUF quantized versions of google/gemma-4-E2B-it created using Intel's AutoRound quantization method.

Quantization Details

The models were quantized using various schemes provided by the auto-round tool. For better compatibility and smaller size, we provide unified multimodal projector (mmproj) files in F16, BF16, and F32 formats.

Files and Sizes

File NameQuant TypeSizeDescription
gemma-4-E2B-it-Q2_K_S.ggufQ2KS2.7 GBExtremely high compression, significant quality loss.
gemma-4-E2B-it-Q2_K_MIXED.ggufQ2KMIXED3.8 GBRecommended high-compression option. Uses Q4 for KV cache with good quality.
gemma-4-E2B-it-Q3_K_S.ggufQ3KS2.9 GBVery high compression, notable quality loss.
gemma-4-E2B-it-Q3_K_M.ggufQ3KM2.9 GBBalanced 3-bit quantization.
gemma-4-E2B-it-Q3_K_L.ggufQ3KL2.9 GBHigh quality 3-bit quantization.
gemma-4-E2B-it-Q4_0.ggufQ4_03.2 GBStandard 4-bit quantization, good balance.
gemma-4-E2B-it-Q4_1.ggufQ4_13.3 GBHigher quality 4-bit quantization than Q4_0.
gemma-4-E2B-it-Q4_K_S.ggufQ4KS3.2 GBSmall 4-bit K-quant, good efficiency.
gemma-4-E2B-it-Q4_K_M.ggufQ4KM3.2 GBRecommended 4-bit K-quant, excellent balance.
gemma-4-E2B-it-Q5_0.ggufQ5_03.4 GBStandard 5-bit quantization, very high quality.
gemma-4-E2B-it-Q5_1.ggufQ5_13.5 GBHigher quality 5-bit quantization than Q5_0.
gemma-4-E2B-it-Q5_K_S.ggufQ5KS3.4 GBSmall 5-bit K-quant, very high quality.
gemma-4-E2B-it-Q5_K_M.ggufQ5KM3.4 GBRecommended 5-bit K-quant, near-lossless.
gemma-4-E2B-it-Q6_K.ggufQ6_K3.6 GB6-bit K-quant, virtually indistinguishable from F16.
gemma-4-E2B-it-Q8_0.ggufQ8_04.7 GB8-bit quantization, near-lossless.
mmproj-model-f16.ggufF16940 MBUnified Projector in Float16 format.
mmproj-model-bf16.ggufBF16942 MBUnified Projector in BFloat16 format.
mmproj-model-f32.ggufF321.8 GBUnified Projector in Float32 format.

Generate the Model

The models were generated using Intel's AutoRound with the following command:

bash
auto-round --model google/gemma-4-E2B-it --output_dir ./quantized/ --scheme <SCHEME> --iters 0

Note: To reproduce this quantization, you need the following Pull Request:

Usage with llama.cpp

These models can be used with llama.cpp. For multimodal usage, you must specify the projector file:

bash
./llama-cli -m gemma-4-E2B-it-Q4_K_M.gguf --mmproj mmproj-model-f16.gguf --image your_image.jpg -p "Describe this image."

About AutoRound

AutoRound is an advanced quantization technique from Intel that aims to minimize accuracy loss through automated rounding optimization.