gbuzhf/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MTP-GGUF

NVIDIA Nemotron 3.5 Lightning 30B A3B — MTP GGUFs

Nemotron-3.5-Lightning carries a native MTP draft head (multi-token prediction, num_nextn_predict_layers: 1) — a full attention + MoE draft block co-trained with the trunk. It is embedded in-model at blk.52 (blk.52.nextn.* is present in every file here), so speculative decoding needs no graft and no extra model.

Quantized from the official llama.cpp-team bf16 master (ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF), whose weights were verified byte-identical to NVIDIA's safetensors before building.

Recipes

Eight tiers in two families, following the KAT-Coder-V2.5-Dev layout:

• —Unsloth Dynamic 2.0 series — UD-IQ4_XS, UD-Q4_K_XL, UD-Q5_K_S. Every non-expert tensor is held at Q8_0; the routed experts carry the tier's bit width.
• —APEX series — APEX-I-Balanced, APEX-I-Quality, APEX-I-Compact, APEX-I-Mini, APEX-I-Compact-v2D-lite. Per-layer precision: the first and last layers are protected, middle layers compressed, shared experts kept high. The v2D-lite variant adds extra headroom on attention keys/values and the output projection.

One thing to know about this model: its dimensions (2688 / 1856 / 3712) are not multiples of 256, so the K-family and IQ-family quant types physically cannot quantize most of the weights — llama.cpp silently substitutes block quants (its own "Q4KM" of this model actually ships Q50/Q80). All tiers therefore use the Q80 / Q50 / Q40 ladder, which is valid on every tensor. **6-bit tiers are not possible for this architecture** (the legacy 6-bit type is not selectable by llama-quantize, and Q6K requires dimensions this model does not have), so the family is eight honest tiers — no tier name overstates its content. The smallest files floor around 19-20 GB. The two exception roles (attention output and SSM output, ne0=4096) take K-quants at the same size, so the imatrix still has something to guide.

All tiers share one imatrix (see below) and the embedded MTP head at Q8_0.

Files

Exact sizes and sha256 are in sha256sums.txt; every file was byte-count verified against the hub after upload.

imatrix

Calibrated on the model's own output (agentic-coding rollouts) blended 1:1 with bartowski's calibration_datav5, evenly interleaved — 980 chunks on the bf16 reference. As with every MTP build: the draft head (blk.52) is never executed during calibration, so it is quantized unguided (here at Q8_0).

Head fingerprint

blk.52.nextn.hnorm.weight  sha256  <filled after build>

Serving

llama-server -m <model>.gguf -c 65536 -fa on --jinja \
  --spec-type draft-mtp,ngram-mod \
  --spec-draft-n-max 1 --spec-draft-n-min 0 --spec-draft-p-min 0.75 \
  --spec-ngram-mod-n-min 8 --spec-ngram-mod-n-max 24 --spec-ngram-mod-n-match 48

draft-mtp uses the embedded head; ngram-mod covers repeat-heavy output. (Requires a llama.cpp build with Nemotron MTP support, e.g. ≥ 2026-08-10.)

Building your own tier

hf download ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF \
  NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.gguf --local-dir .
llama-quantize --imatrix <model>-imatrix.gguf \
  --tensor-type-file <tier-map>.txt master.gguf out.gguf <BASE>

Credits

NVIDIA — Nemotron-3.5-Lightning-30B-A3B · llama.cpp team — official bf16 master · Unsloth — Dynamic 2.0 quant map logic · mudler — APEX method · bartowski — calibration corpus and the arch reference imatrix · llama.cpp

License: openmdw-1.1, inherited from the base model.