CoolFace
Modelpublic

lmcoleman/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MagicQuant-GGUF

sourceHugging Faceotherupdated 1mo agoView on Hugging Face
0likes333downloads
Model Card

NVIDIA-Nemotron-3.5-Lightning-30B-A3B — MagicQuant hybrid GGUFs

Per-group hybrid quantizations of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 (hybrid-SSM MoE, ~30B total / A3B active), produced by MagicQuant's measured evolutionary search: tensors are grouped by role, candidate per-group scheme assignments are actually rendered and perplexity-measured (not predicted), and winners are selected per size tier. imatrix-calibrated, KL-scored (kl_weight 0.1), 15 measured candidates over 3 rounds.

Pins: MagicQuant 14e9d62, Foundry 2f99202. Search ledger and the full pre-registered comparison record live with the run.

Files

filesizePPL (wikitext-2, ctx 512, full corpus)vs BF16 (6.83)
...-Q4_K_M.gguf17.60 GiB6.8948 ± 0.046+0.95%
...-Q5_K_M.gguf20.60 GiB6.8579+0.41%
...-Q6_K.gguf24.13 GiB6.7979−0.47% (within noise of baseline)

All PPLs from one CPU llama.cpp instrument (ctx 512, full wiki.test.raw, same baseline). Q4 was re-measured standalone with error bars; Q5/Q6 values are the search's internal measurements on the identical instrument (the standalone Q4 re-measurement reproduced the search-internal value to 4 decimals).

Tier filenames follow effective-bpw bands, not a single uniform quant type — each file mixes schemes per group (e.g. the Q4 winner spends its budget as D:Q4_K_M E:Q8_0 H:BF16 K:BF16 O:BF16 Q:Q8_0 S:Q8_0 U:IQ4_NL X:MXFP4_MOE).

How the Q4 winner was validated

As a controlled experiment, an alternative per-tensor budget-knapsack allocation (MagicQuant v2, which predicts damage from a distortion table instead of measuring candidates) was built at exactly the Q4 winner's size, under identical pinned code, and measured on the same instrument:

At matched 17.6 GiB, the measured per-group search beat the predicted per-tensor allocation: +0.95% vs +3.47% quality loss — a +2.50% PPL gap, far outside the measurement error.

The losing artifact is not published (dominated at identical size); the numbers above are from the pre-registered comparison record.

Notes

  • —Chat template embedded and verified present in every file before upload.
  • —SSM/state tensors are held at F32/F16 per llama.cpp requirements (the never-quantize and f32-required-operand classes); BF16-designated groups are written as F16 on disk (llama.cpp BF16 compute-graph limitation).
  • —Requires a llama.cpp recent enough for this architecture and for MXFP4MOE/IQ4NL tensor types (mid-2026 or newer).
  • —Built and measured on an AMD Strix Halo (Ryzen AI MAX+ 395, gfx1151) host.