CoolFace
Modelpublic

leonsarmiento/Nemotron-3.5-Lightning-30B-A3B-6bit-XL-mlx

sourceHugging Faceotherupdated 18d agoView on Hugging Face
0likes304downloads
Model Card

Nemotron-3.5-Lightning-30B-A3B — 6-bit XL (MLX)

ℹ️ MTP note: The source model ships a NextN/MTP prediction layer (num_nextn_predict_layers: 1), but this MLX build does not include MTP weights — the layer is dropped during conversion (52 backbone layers, no MTP tensors). No MTP speculative decoding on any engine; the residual MTP fields in config.json are inert.

MLX 6/8-bit BaseQuant_XL quantization of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 for Apple Silicon inference.

About XL Quantization

BaseQuant_XL is a data-agnostic static mixed-precision recipe — no calibration data, no iMatrix, no activation-weighted sampling. It allocates precision by layer importance using a simple heuristic: routing-critical layers stay at full precision, every-token layers get 8-bit (near-lossless), and sparse MoE experts get 6-bit (natural redundancy from 128 experts, only 6 active per token). This contrasts with data-dependent methods (iMatrix, AWQ, GPTQ, oQ, oQ4e) that can skew representation toward well-represented domains.

Quantization Recipe (BaseQuant_XL 6/8)

PrecisionLayersRationale
bf16lm_head, shared expertRouting/output-critical — errors here are qualitatively different
8-bitembeddings, attention (q/k/v/oproj), Mamba2 (inproj, out_proj, conv1d)Every-token layers — 8-bit is near-lossless
6-bitrouted experts (128 experts × 23 MoE layers + MTP MoE)Bulk parameters — natural redundancy tolerates lower precision

The MoE router gate (gate.weight, e_score_correction_bias) is kept in full precision as bare arrays (not quantizable layers).

  • —6.834 bits per weight, ~25 GB (6 shards)
  • —Group size: 64

Model Architecture

Nemotron-H — hybrid Mamba2 + Transformer + MoE architecture:

  • —52 layers: 23 Mamba2 (SSM), 6 Attention, 23 MoE
  • —1 NextN (MTP) prediction layer: [attention, moe] — Multi-Token Prediction for speculative decoding
  • —30B total parameters, ~3B active per token (6 of 128 experts active)
  • —1 shared expert with shared_expert_overlap: true
  • —mlp_hidden_act: relu2, routed_scaling_factor: 2.5
  • —tie_word_embeddings: false
  • —Context length: 262,144 tokens
  • —ChatML-style template with <think> reasoning and tool-call support

What Makes Lightning Different

  • —Mamba-2 hybrid architecture — interleaves SSM layers with MoE/Attention for higher throughput and lower memory than pure Transformer
  • —Multi-Token Prediction (MTP) — trained with MTP layers that predict multiple future tokens, enabling self-speculative decoding
  • —3B active / 30B total — efficient for single-device deployment

Inference Parameters

temperature: 1.0
top_p: 0.95
top_k: 40
min_p: 0.01
repeat_penalty: 1.05
reasoning_parser: nemotron_v3
tool_call_parser: qwen3_coder

Thinking mode is controlled via the chat template kwarg enable_thinking (default: true).

Usage

python
from mlx_lm import load, generate

model, tokenizer = load("leonsarmiento/Nemotron-3.5-Lightning-30B-A3B-6bit-XL-mlx")

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Explain quantum entanglement."}],
    add_generation_prompt=True,
)

response = generate(model, tokenizer, prompt=prompt, max_tokens=512)
print(response)

Also compatible with LM Studio and oMLX — point it at the model directory.