CoolFace
Modelpublic

mlx-community/gemma-4-26B-A4B-it-OptiQ-4bit

sourceHugging Facegemmaupdated 2mo agoView on Hugging Face
20likes1.4kdownloads
Model Card

mlx-community/gemma-4-26B-A4B-it-OptiQ-4bit

Built with [mlx-optiq](https://mlx-optiq.com), the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. Try the Lab · All OptiQ quants · Docs

A 4-bit mixed-precision MLX quant produced by mlx-optiq, the sensitivity-aware quantization toolkit for Apple Silicon. Beats stock uniform 4-bit on every benchmark in the six-metric Capability Score.

A 4-bit mixed-precision MLX quant of google/gemma-4-26B-A4B-it. Per-layer bit-widths come from a KL-divergence sensitivity pass on a six-domain calibration mix (prose · reasoning · code · agent · tool-call · constraint-bearing instructions). Sensitive layers go to 8-bit; robust ones stay at 4-bit. The on-disk size is within ~5 % of a stock uniform 4-bit MLX quant.

Quantization details

PropertyValue
Predominant precision4-bit
Layers at 8-bit (sensitive)246
Layers at 4-bit (robust)79
Total quantized layers325
Group size64
Calibration mixsix-domain mix (40 samples × 6 domains)
Reference for sensitivitybf16 (auto-resolved; falls back to uniform-4-bit if bf16 doesn't fit)
Speculative drafterserved with `mlx-community/gemma-4-26B-A4B-it-assistant-bf16` via optiq serve --drafter

We follow the same naming convention llama.cpp uses for Q4KM and similar mixed-precision quants: the "4-bit" label is for the predominant precision, not the weighted average. The mixed allocation is what lets this build beat stock uniform-4-bit on every benchmark below at the same disk size.

Usage

Load it with mlx-lm and use it as usual:

bash
pip install mlx-lm
python
from mlx_lm import load, generate

model, tokenizer = load("mlx-community/gemma-4-26B-A4B-it-OptiQ-4bit")
response = generate(
    model, tokenizer,
    prompt="Explain quantum computing in simple terms.",
    max_tokens=200,
)

For more (mixed-precision KV-cache serving, sensitivity-aware LoRA fine-tuning, OpenAI + Anthropic-compatible inference server, hot-swap mounted adapters, sandboxed Python execution for agent workflows), install `mlx-optiq`:

bash
pip install mlx-optiq

Speculative decoding (assistant drafter)

Gemma-4 ships a separate small drafter for speculative decoding. Pair this quant with `mlx-community/gemma-4-26B-A4B-it-assistant-bf16` for faster decode:

bash
optiq serve --model mlx-community/gemma-4-26B-A4B-it-OptiQ-4bit \
            --drafter mlx-community/gemma-4-26B-A4B-it-assistant-bf16

See the Gemma-4 family guide on mlx-optiq.com for sampling defaults, training recipes, and family-specific caveats.

Benchmarks

Six-metric Capability Score (mean of MMLU + GSM8K + IFEval + BFCL + HumanEval + HashHop). Apples-to-apples comparison against stock uniform 4-bit:

MetricOptiQUniform 4-bitΔ
MMLU (5-shot, 1000 samples)65.0%61.1%+3.9
GSM8K (1000 samples, 3-shot CoT)93.8%91.7%+2.1
IFEval (full set, strict)73.0%74.5%-1.5
BFCL-V3 simple (200 calls)91.5%92.0%-0.5
HumanEval (164 problems, pass@1)90.2%88.4%+1.8
HashHop (long-context retrieval)41.0%30.0%+11.0
Capability Score (mean of 6)75.7672.95+2.81
KL vs uniform-4-bit reference (mean / p95)1.1874 / 5.2286,,
On-disk size16.4 GB14.5 GB+1.9

Every metric gets one equal vote. Disk size is reported next to the score as an honest second axis instead of being folded into the score. See the eval-framework writeup for the full methodology.

Links

Quantize your own

This quant was produced by mlx-optiq. Point it at any Hugging Face model to get the same sensitivity-aware mixed precision:

bash
pip install mlx-optiq
optiq convert <hf-model-id> --target-bpw 5.0 --candidate-bits 4,8
optiq lab   # full local workbench: chat, compare, quantize, fine-tune

License

Gemma license (inherits from base model). See https://ai.google.dev/gemma/terms for the terms of use.