Modelpublic

chantra/gemma-4-4bit-it-mlx-dwq-ram-5gb

sourceHugging Faceupdated 5mo agoView on Hugging Face

0likes54downloads

Model Card

Gemma 4 E4B IT — DWQ 4-bit (MLX, ~5 GB RAM)

4-bit DWQ (Distillation + Weight Quantization) of gemma-4-E4B-it for Apple Silicon via MLX. Quantized from an 8-bit teacher. Fits comfortably in 5 GB RAM with better perplexity than standard 4-bit.

Model Details

Property	Value
Base model	`google/gemma-4-e4b-it`
Parameters	7.5B
Quantization	DWQ 4-bit (teacher: 8-bit)
Framework	MLX
RAM	~5 GB

Evaluation Results

ARC Easy

Metric	Score
Accuracy	0.7113
Accuracy (normalized)	0.6540

Perplexity

—Dataset: allenai/tulu-3-sft-mixture (256 samples, 131,072 tokens, seq len 512)
—Perplexity: 42.847 ± 0.638
—Speed: 2002 tokens/sec
—Time: 65.35s
—Peak Memory: 11.25 GB

TruthfulQA

Generation:

Metric	Score
BLEU Max	9.69
BLEU Acc	0.52
ROUGE-1 Acc	0.56
ROUGE-2 Acc	0.44
ROUGE-L Acc	0.55

Multiple Choice:

Metric	Score
MC1 Accuracy	0.31
MC2 Accuracy	0.5637

Comparison vs Other Quantizations

Model	ARC Easy (acc)	Perplexity ↓	TruthfulQA MC2	RAM
8-bit (base)	0.7168	44.44	0.5705	8.5 GB
DWQ 4-bit (this)	0.7113	42.85 ✅	0.5637	4.8 GB
Std 4-bit	0.7075	54.07 ❌	0.5506	4.8 GB

DWQ 4-bit achieves lower perplexity than 8-bit while cutting memory by ~4 GB. Standard 4-bit shows significant perplexity degradation (54.07) in comparison.

Usage

python

from mlx_lm import load, generate

model, tokenizer = load("chantra/gemma-4-4bit-it-mlx-dwq-ram-5gb")
response = generate(model, tokenizer, prompt="Hello!", max_tokens=512)
print(response)