mlx-community/NVIDIA-Nemotron-3-Nano-30B-A3B-OptiQ-4bit

mlx-community/NVIDIA-Nemotron-3-Nano-30B-A3B-OptiQ-4bit

Built with [mlx-optiq](https://mlx-optiq.com), the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. Try the Lab · All OptiQ quants · Docs

A 4-bit mixed-precision MLX quant of mlx-community/NVIDIA-Nemotron-3-Nano-30B-A3B-MLX-BF16 produced by mlx-optiq, the sensitivity-aware quantization toolkit for Apple Silicon. +2.0 Capability Score over stock uniform 4-bit, winning or tying every one of the six benchmarks.

Nemotron 3 Nano 30B-A3B is a hybrid Mamba2 + attention model with a 128-expert sparse MoE (≈3B active parameters per token). OptiQ measures each linear's KL-divergence sensitivity against a reference forward pass and assigns 4-bit or 8-bit per-layer, including the fused switch_mlp routed-expert tensors that dominate the model's parameter mass. Sensitive layers go to 8-bit; robust ones (including most of the experts) stay at 4-bit.

Quantization details

We follow the same naming convention llama.cpp uses for Q4KM-style mixed-precision quants: the "4-bit" label is for the predominant precision, not the weighted average. Most of the 8-bit layers are the small mamba / attention projections; the big routed-expert tensors mostly stay at 4-bit, which is how the model lands at 5.05 BPW.

Usage

Load it with mlx-lm (the custom NemotronH modeling files ship in the repo and are picked up automatically):

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/NVIDIA-Nemotron-3-Nano-30B-A3B-OptiQ-4bit")
response = generate(
    model, tokenizer,
    prompt="Explain how a sparse mixture-of-experts router decides which experts to activate.",
    max_tokens=400,
)

For mixed-precision KV-cache serving and sensitivity-aware LoRA fine-tuning, install `mlx-optiq`:

pip install mlx-optiq

# Serve with the bundled KV-cache recipe
optiq serve --model mlx-community/NVIDIA-Nemotron-3-Nano-30B-A3B-OptiQ-4bit \
            --kv-config kv_config.json

Benchmarks

Six-metric Capability Score (mean of MMLU + GSM8K + IFEval + BFCL + HumanEval + HashHop). Apples-to-apples comparison against stock uniform 4-bit:

OptiQ wins or ties every benchmark. The mixed-precision allocation costs ~4 GB more on disk than stock uniform 4-bit, that disk buys a clean sweep across math, code, instruction-following, and long-context retrieval. Every metric gets one equal vote; disk size is reported next to the score as an honest second axis instead of being folded in. See the eval-framework writeup for the full methodology.

Links

• —Project website: mlx-optiq.com
• —PyPI: pypi.org/project/mlx-optiq
• —Calibration mix: mlx-optiq.com/blog/calibration-mix
• —Eval framework: mlx-optiq.com/blog/eval-framework
• —Base model: mlx-community/NVIDIA-Nemotron-3-Nano-30B-A3B-MLX-BF16

Base model

This is a quantized derivative of NVIDIA Nemotron 3 Nano 30B-A3B. See the NVIDIA Open Model License for terms, the quant is distributed under the same license as the base.

Quantize your own

This quant was produced by mlx-optiq. Point it at any Hugging Face model to get the same sensitivity-aware mixed precision:

pip install mlx-optiq
optiq convert <hf-model-id> --target-bpw 5.0 --candidate-bits 4,8
optiq lab   # full local workbench: chat, compare, quantize, fine-tune