CoolFace
Modelpublic

Vontra/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-6bit

sourceHugging Faceotherupdated 19d agoView on Hugging Face
2likes512downloads
Model Card

<p align="center"> <img src="https://img.shields.io/badge/NVIDIA-Nemotron-76B900?style=for-the-badge&logo=nvidia&logoColor=white" alt="NVIDIA Nemotron"> <img src="https://img.shields.io/badge/Apple_Silicon-MLX-000000?style=for-the-badge&logo=apple&logoColor=white" alt="Apple silicon MLX"> <img src="https://img.shields.io/badge/Vontra-oMLX-6E56CF?style=for-the-badge&logo=huggingface&logoColor=white" alt="Vontra oMLX"> </p>

<h1 align="center">NVIDIA Nemotron 3.5 Lightning 30B-A3B — MLX 6-bit</h1>

<p align="center"> A native Apple-silicon conversion of <a href="https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16">nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16</a>, quantized with stock 6-bit affine weights and packaged for MLX-LM and oMLX. </p>

<p align="center"> <a href="https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16">Original model</a> · <a href="https://developer.nvidia.com/nemotron">NVIDIA Nemotron</a> · <a href="https://github.com/ml-explore/mlx-lm">MLX-LM</a> · <a href="https://openmdw.ai/license/1-1/">OpenMDW 1.1 license</a> </p>

About this conversion

This repository contains a stock 6-bit affine MLX conversion of NVIDIA Nemotron 3.5 Lightning. The source is a 30B-total / 3B-active hybrid mixture-of-experts model that interleaves Mamba-2, sparse MoE, and attention layers. The upstream tokenizer, chat template, and generation configuration are preserved.

ItemValue
Base model`nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16`
FormatMLX safetensors
Quantization6-bit affine, group size 64
Conversion stackmlx-lm 0.31.3, mlx 0.32.0
Weight shards5
Weight size25.67 GB (23.90 GiB)
Maximum configured context262,144 tokens
Architecturenemotron_h — Mamba-2 + sparse MoE + attention
[!NOTE] MLX-LM reported an effective precision of 6.503 bits per weight.

Apple-silicon performance

This checkpoint was load-tested and generation-tested on the following machine:

HardwareConfiguration
HostMac Studio
ChipApple M3 Ultra
CPU32 cores (24 performance + 8 efficiency)
Unified memory256 GB
RuntimeMLX-LM 0.31.3 / MLX 0.32.0

A warmed local test produced:

MeasurementResult
Decode (median)138.38 tokens/s
Reported peak memory25.84 GB
Timed runs3 × 256 generated tokens
Warm-up32 generated tokens
Prompt36 tokens after chat templating

The decode figure is the median of three greedy 256-token runs after a 32-token Metal-kernel warm-up. It is a practical local reference, not a controlled cross-platform benchmark. Prompt length, context growth, sampler settings, memory pressure, thermal state, and MLX/oMLX versions can materially change performance.

Quick start with MLX-LM

Install recent MLX-LM and Hugging Face tooling:

bash
python -m pip install -U mlx-lm huggingface_hub

Run directly from the Hub:

bash
mlx_lm.generate \
  --model Vontra/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-6bit \
  --prompt "Explain why hybrid Mamba and MoE architectures are efficient." \
  --max-tokens 512 \
  --temp 1.0 \
  --top-p 0.95

Reasoning mode is enabled by the upstream chat template by default. To disable it:

bash
mlx_lm.generate \
  --model Vontra/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-6bit \
  --chat-template-config '{"enable_thinking": false}' \
  --prompt "Write a short hello-world program in Swift." \
  --max-tokens 256

Python usage:

python
from mlx_lm import load, generate


model, tokenizer = load("Vontra/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-6bit")
messages = [
    {"role": "user", "content": "Explain sparse mixture-of-experts routing."}
]
prompt = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=False,
    enable_thinking=False,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=512))

To download the repository first:

bash
hf download Vontra/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-6bit \
  --local-dir ~/.omlx/models/Vontra/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-6bit

Using it with oMLX

  1. 1.Place the downloaded model at ~/.omlx/models/Vontra/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-6bit.
  2. 2.Refresh the oMLX model registry.
  3. 3.Load NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-6bit and use the normal chat UI or OpenAI-compatible endpoint.

Example request:

bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OMLX_API_KEY" \
  -d '{
    "model": "NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-6bit",
    "messages": [{"role": "user", "content": "Say hello from Nemotron on MLX."}],
    "temperature": 1.0,
    "top_p": 0.95,
    "max_tokens": 128
  }'

For long prompts, begin with a conservative context limit and increase it while watching memory pressure. The configured 256K context is a model capability, not a guarantee that every host can prefill that context within its available unified memory.

Architecture

Nemotron 3.5 Lightning is a hybrid sparse model designed for efficient agentic and reasoning workloads.

Architecture detailUpstream value
Total / active parameters30B / 3B
Layers52
Routed / shared experts128 / 1
Active routed experts6
Attention heads / KV heads32 / 2
Hidden size2,688
Expert intermediate size1,856
Vocabulary size131,072
Configured context262,144 tokens

The upstream release is intended for coding, tool use, reasoning, research, and customization. For NVIDIA's evaluations, deployment guidance, intended use, limitations, safety information, and full architecture discussion, see the original model card.

Conversion and validation notes

  • —Source weights: NVIDIA's BF16 checkpoint.
  • —Quantization group size: 64.
  • —Quantization mode: affine.
  • —The upstream chat_template.jinja is preserved.
  • —All 729 converted tensors and every indexed shard were checked locally.
  • —The model was loaded and exercised through end-to-end generation on Apple silicon.
  • —Quantization can reduce output quality relative to BF16; use a higher-precision variant when quality matters more than memory use.

This is a community conversion, not an official NVIDIA release. Validate quality and numerical behavior on your own representative workload before production use.

License and attribution

The upstream model is released under the OpenMDW License Agreement, version 1.1. A copy is included in this repository; review it before use or redistribution.

All model design, training, benchmark, and upstream documentation credit belongs to NVIDIA and the original contributors. The MLX conversion, Apple-silicon validation, and packaging are provided by Vontra.

<!-- vontra-chooser-start -->

Choose for your Mac

64GB Macs · 128GB Macs · 256GB Macs

Published peak memory: 25.84 GB; estimated starting tier: 64GB, leaving about 38 GB nominal headroom. The collections use published M3 Studio peaks with at least 25% nominal headroom; fit on other Macs is an estimate, and full context is not guaranteed. Start with short context and one request.

Runtime and evidence

The exact tested oMLX application version is not recorded here; a library version is not an app version. The original performance tables retain their benchmark conditions and speed figures; this documentation update adds no new test results.

Quick start and demo prompt

bash
hf download Vontra/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-6bit --local-dir ./models/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-6bit

Add the downloaded folder to oMLX model directories, refresh the list, and follow this card's architecture and MTP compatibility requirements before loading.

Try this in a new chat with a 128-token output limit:

text
Explain why the sky looks blue in three short sentences.

This is a demo prompt to try, not a recorded successful run; a captured demonstration for this documentation update is not yet available.

Follow Vontra for new Apple Silicon releases and fixes. <!-- vontra-chooser-end -->