CoolFace
Modelpublic

sphaela/gemma-4-E4B-it-AutoRound-GGUF

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face
1likes406downloads
Model Card

Gemma-4-E4B-it GGUF (AutoRound Quantized)

This repository contains GGUF quantized versions of google/gemma-4-E4B-it created using Intel's AutoRound quantization method.

Quantization Details

The models were quantized using various schemes provided by the auto-round tool. For better compatibility and smaller size, we provide unified multimodal projector (mmproj) files in F16, BF16, and F32 formats.

Files and Sizes

File NameQuant TypeSizeDescription
gemma-4-E4B-it-Q2_K_S.ggufQ2KS4.0 GBExtremely high compression, significant quality loss.
gemma-4-E4B-it-Q2_K_MIXED.ggufQ2KMIXED5.6 GBRecommended high-compression option. Uses Q4 for KV cache with good quality.
gemma-4-E4B-it-Q3_K_S.ggufQ3KS4.3 GBVery high compression, notable quality loss.
gemma-4-E4B-it-Q3_K_M.ggufQ3KM4.3 GBBalanced 3-bit quantization.
gemma-4-E4B-it-Q3_K_L.ggufQ3KL4.3 GBHigh quality 3-bit quantization.
gemma-4-E4B-it-Q4_0.ggufQ4_04.8 GBStandard 4-bit quantization, good balance.
gemma-4-E4B-it-Q4_1.ggufQ4_15.1 GBHigher quality 4-bit quantization than Q4_0.
gemma-4-E4B-it-Q4_K_S.ggufQ4KS4.8 GBSmall 4-bit K-quant, good efficiency.
gemma-4-E4B-it-Q4_K_M.ggufQ4KM4.8 GBRecommended 4-bit K-quant, excellent balance.
`gemma-4-E4B-it-Q5_0.gguf"Q5_05.3 GBStandard 5-bit quantization, very high quality.
gemma-4-E4B-it-Q5_1.ggufQ5_15.5 GBHigher quality 5-bit quantization than Q5_0.
gemma-4-E4B-it-Q5_K_S.ggufQ5KS5.3 GBSmall 5-bit K-quant, very high quality.
gemma-4-E4B-it-Q5_K_M.ggufQ5KM5.3 GBRecommended 5-bit K-quant, near-lossless.
gemma-4-E4B-it-Q6_K.ggufQ6_K5.8 GB6-bit K-quant, virtually indistinguishable from F16.
gemma-4-E4B-it-Q8_0.ggufQ8_07.5 GB8-bit quantization, near-lossless.
mmproj-model-f16.ggufF16945 MBUnified Projector in Float16 format.
mmproj-model-bf16.ggufBF16946 MBUnified Projector in BFloat16 format.
mmproj-model-f32.ggufF321.8 GBUnified Projector in Float32 format.

Generate the Model

The models were generated using Intel's AutoRound with the following command:

bash
auto-round --model google/gemma-4-E4B-it --output_dir ./quantized/ --scheme <SCHEME> --iters 0

Note: To reproduce this quantization, you need the following Pull Request:

Usage with llama.cpp

These models can be used with llama.cpp. For multimodal usage, you must specify the projector file:

bash
./llama-cli -m gemma-4-E4B-it-Q4_K_M.gguf --mmproj mmproj-model-f16.gguf --image your_image.jpg -p "Describe this image."

About AutoRound

AutoRound is an advanced quantization technique from Intel that aims to minimize accuracy loss through automated rounding optimization.