Modelpublic

hexoy/gemma-4-e2b-distilled

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes7downloads

Model Card

Gemma 4 E2B Distilled

Distilled Gemma 4 E2B with all 35 language-model MLPs replaced by two-factor Monarch maps. The released BF16 model has 3,682,268,704 parameters, instead of 5,104,297,504 in the original Gemma 4 E2B.

1. Weight Storage

Model	Status	Loaded weights	Serialized weights	Reduction vs dense BF16
Dense Gemma 4 BF16	Reference	9.507 GiB	9.543 GiB	-
Distilled Gemma 4 BF16	Released	6.859 GiB	6.859 GiB	27.86%
Distilled Gemma 4 + LoRA r8 BF16	Experimental	6.876 GiB	6.877 GiB	27.68%
Distilled Gemma 4 + INT8 linears	Released	6.135 GiB	6.136 GiB	35.48%

BF16 sizes use two bytes per parameter; serialized LoRA size was audited from the two safetensor shards. Values exclude activations and CUDA workspaces.

2. TinyHellaSwag

All rows used the same RTX PRO 6000, fixed batch size 32, seed 1234, 100 official examples, 10-shot prompts, and no chat template.

Model	Status	GP-IRT accuracy	Raw accuracy	Runtime	Peak VRAM
Dense Gemma 4 BF16	Reference	39.23%	29%	28.50 s	57.88 GiB
Distilled Gemma 4 BF16	Released	32.35%	22%	23.33 s	55.23 GiB
Distilled Gemma 4 + LoRA r8 BF16	Experimental	33.25%	23%	24.93 s	55.25 GiB
Distilled Gemma 4 + INT8 linears	Released	30.57%	21%	23.14 s	54.51 GiB

Rank-8 LoRA improved the 35-layer source by 0.90 GP-IRT points and one correct item, but missed the former +1.0 point or +2 item release gate. The 100-example benchmark is noisy.

Usage

python

from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "hexoy/gemma-4-e2b-distilled"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype="auto",
    device_map="auto",
)

The rank-8 result is published separately at `hexoy/gemma-4-e2b-monarch-35mlp-lora-r8`. It is a benchmark-recovery experiment: corrected 64-token generation repeated phrases for text and remained malformed or inaccurate for images. It is not a general text or multimodal recovery release.

Reproducibility

—BF16 model revision: f897353fca328b1cc5fd2e12d645773ca637f5f0
—GitHub repository: `ratmir-miftachov/gemma-distillation`
—INT8 variant: `hexoy/gemma-4-e2b-monarch-35mlp-int8`
—Experimental LoRA r8 variant: `hexoy/gemma-4-e2b-monarch-35mlp-lora-r8`
—LoRA source commit: 1435571b20dd26c073a535678975884154add5b8

Detailed training and release evidence is retained privately.

Derived from google/gemma-4-E2B-it. See NOTICE for the modification summary.