jokernifty/gemma-4-12B-it-mlx-4bit

gemma-4-12B-it (MLX, 4-bit quantized, text-only)

4-bit MLX conversion of google/gemma-4-12B-it.

Note: Gemma 4 12B is a unified encoder-free multimodal model (text + image + audio). This conversion is text-only — the vision and audio embedder weights are stripped because mlx-lm doesn't yet handle the gemma4_unified encoder-free multimodal path. The text backbone is intact.

Conversion details

• —4-bit, group size 64
• —Token embedding kept in bf16 (~2 GB) — quantizing 262K × 3840 in one op exceeded the macOS Metal command-buffer watchdog
• —Total size: ~8.1 GB across 5 shards
• —Per-layer quantization to dodge GPU timeouts

Quick start

This model uses the gemma4_unified modeltype, which isn't registered in `mlx-lm` yet. Until upstream adds it, drop this 4-line alias into your `mlxlm/models/` directory:

# mlx_lm/models/gemma4_unified.py
from . import gemma4
ModelArgs = gemma4.ModelArgs

_SKIP = ("vision_embedder.", "audio_embedder.", "audio_input.", "vision_input.",
         "embed_vision_tokens.", "embed_audio_tokens.", "embed_video_tokens.")

class Model(gemma4.Model):
    def sanitize(self, weights):
        f = {k:v for k,v in weights.items()
             if not any((k[len("model."):] if k.startswith("model.") else k).startswith(p) for p in _SKIP)}
        return super().sanitize(f)

Then:

python -m mlx_lm chat --model jokernifty/gemma-4-12B-it-mlx-4bit

Recommended sampling (from Google's model card)

• —temperature = 1.0, top_p = 0.95, top_k = 64
• —Enable thinking mode by prepending <|think|> to the system prompt.

License

Apache 2.0 (inherited). Use is also subject to the Gemma Terms and Prohibited Use Policy. All credit for the model goes to Google DeepMind.