mattbucci/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-AWQ

Nemotron-3-Nano-Omni-30B-A3B-Reasoning-AWQ

AWQ 4-bit (group_size=64) quantization of **nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning**, targeting 2× RTX 3090 (48 GB) and similar 40–48 GB inference fleets. Verified serving + 6/6 capabilities at full 262144 context on SGLang v0.5.13.post1.

What's quantized vs passthrough

• —INT4 W4A16 (GPTQ-calibrated, 5934 tensors): all language_model.backbone.layers.*.mixer.experts.*.{up_proj,down_proj}.weight MoE expert linears, plus shared_experts.{up_proj,down_proj}.weight. NemotronH's MoE is non-gated (squared-ReLU activation, up_proj only — no separate gate_proj), so the per-expert weight count is 2× per layer, not 3× as in a standard SwiGLU MoE.
• —BF16 passthrough: 23 Mamba2 SSM layers (in_proj, out_proj, conv1d, dt_bias, A_log, D, RMSNorms), 6 attention layers (q_proj, k_proj, v_proj, o_proj), all router gates, all mlp projector heads (mlp1), embeddings + LM head, the CRADIO v4-H vision tower (vision_model) in full, and the Parakeet audio tower (sound_encoder + sound_projection) in full.

The quantization_config.modules_to_not_convert and .ignore fields in config.json are populated with the exact 14-entry exclusion list — the AWQ loader must see this to keep the BF16 passthrough modules unquantized. (Earlier ships had this empty, causing SGLang to attempt AWQ-wrapping the Mamba in_proj and crashing on shape constraints — see Changelog v3.)

Calibration recipe

omni_thinking_tools recipe, 1024 samples × 2048 tokens (~2.1 M cal tokens). Eight-slice mix that covers the model's live token routes:

`drop_images=True` is used in rows_to_text — image/audio/video bytes are dropped before tokenization. The language-model submodule sees text + the model's own modality placeholder tokens, mirroring inference where the CRADIO / Parakeet towers produce their own embeddings into the LM's input stream at those positions. This is the same text-with-placeholders approximation used for the rest of our multimodal AWQ ships (Qwen3-VL, gemma-4, Devstral-vision).

Pipeline: llmcompressor.oneshot(model=model.language_model, …) targeting the inner NemotronHForCausalLM only (the multimodal wrapper's image_flags.squeeze(-1) AttributeErrors on text-only rows; the encoders + Omni wrapper remain attached so the saved checkpoint still ships them). GPTQModifier(scheme="W4A16", targets="Linear", offload_hessians=True, bypass_divisibility_checks=True), group_size=64, moe_calibrate_all_experts=True (default in llmcompressor ≥0.11 — forces every token through every expert during cal so rare-routed experts don't get degenerate scales).

Wall clock: Phase 2 calibration ran 9 days 6 h 7 min end-to-end on a single CPU host (no GPU; Hessian build is GPU-light; expert quantization dominates).

Audit (check_awq_scales.py --base)

Scanned *5934 `.scales** and **5934 .qweight` tensors against the BF16 base (dead-channel comparator enabled). 0 flagged.* No all-zero scales, no majority-zero qweights, no NaN/Inf, no live-base / zero-scale mismatches.

This is the structural gate that catches the v2 "scales=0 NaN logits" and v3 "qweight=0 rare-expert-under-cal" failure modes we've seen on other MoE quantizations. Both clean here.

Per-layer GPTQ error pattern (audit-only, not a quality gate)

Elevated down_proj and up_proj errors were observed during calibration on later MoE layers — most prominent on layer 47 (peak expert.99.down_proj error 2311), layer 49 (sustained 400–900 across mid-rank experts), layer 34 (range 384–689), and layer 27 (range 65–347). The structural audit above is clean, so these are quality-signal-only flags, not defects. Live capability validation (6/6 PASS — see below) and 256K decode benchmarking both succeed, indicating the elevated errors did not translate into observable inference defects.

Inference

SGLang (verified working — 2×RTX 3090, full 262144 ctx)

This artifact serves end-to-end on SGLang v0.5.13.post1 + two open-source-friendly patches plus the standard nemotron3-omni launcher preset:

# scripts/launch.sh nemotron3-omni on the v0.5.13.post1 tree
# QUANT=moe_wna16, TP=2, --reasoning-parser nemotron_3 --tool-call-parser qwen3_coder
# --trust-remote-code, MambaRadixCache hybrid_ssm=True, fp8 KV
# Boots at full 262144 with max_total_num_tokens=5250973 (5.25 M-tok KV pool,
# ~20× the context — only 6 of 52 layers are attention; Mamba layers use
# the SSM-state cache, not the regular KV pool).

Required patches (both backend-independent, both upstream-PR-worthy):

Build receipts in `mattbucci/2x-3090-GA102-300-A1-sglang-inference` — see patches/052-*.patch, patches/053-*.patch, and benchmarks/nemotron3-omni-awq-serve-2026-06-16.md for the full serving recipe.

Also required in the env: librosa==0.11.0 (the Parakeet audio extractor imports it).

6-modality capability validation (validate_capabilities.py)

All six capabilities pass on the served model (receipt: benchmarks/nemotron3-omni-v0513-caps.json):

Server stayed alive through every modality. Zero reshape / EVS errors.

256K decode performance (single-user M=1, cuda-graph ON)

~5% decay over 250× depth. NemotronH's hybrid is 23 Mamba layers (O(1) recurrent state) + 6 attention layers — so deep-decode bandwidth is dominated by the int4 expert weights, not by KV reads. Beats the R9700 FP8 reference (74.79 short / 49.22 @ 230 K) on both ends. Prefill is also cheap at depth: TTFT ~1.1 s @ 200 K.

Open lever for further wins: the int4 MoE currently runs an untuned default Triton config (E=128,N=464,…int4_w4a16.json not yet supplied). A tuned fused-MoE config would lift decode further.

transformers (CPU/MPS or single-GPU fp16)

from transformers import AutoModelForCausalLM, AutoProcessor

model = AutoModelForCausalLM.from_pretrained(
    "mattbucci/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-AWQ",
    trust_remote_code=True,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(
    "mattbucci/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-AWQ",
    trust_remote_code=True,
)

The custom modeling files (modeling.py, modeling_nemotron_h.py, configuration*.py, audio_model.py, etc.) are bundled — trust_remote_code=True resolves the architecture via auto_map.

Recommended sampler

(Identical to upstream — preserved through quantization.)

License & attribution

Inherits the NVIDIA Open Model Agreement from the upstream base — the model is available for commercial use. See the agreement for full terms.

This quantization derivative is published non-commercially by mattbucci for research/community use; downstream users are bound by the same upstream license.

Upstream attribution:

NVIDIA Nemotron 3 Nano Omni is a multimodal LLM unifying video, audio, image, and text understanding for enterprise Q&A, summarization, transcription, and document intelligence. Developed by NVIDIA as part of the Nemotron model family. Improved using Qwen3-VL-30B-A3B-Instruct, Qwen3.5-122B-A10B, Qwen3.5-397B-A17B, Qwen2.5-VL-72B-Instruct, and gpt-oss-120b — see the upstream training-data section.

Provenance

• —Quantization rig: 2× RTX 3090 (GA102), 92 GB RAM + 64 GB swap, Arch Linux, miniforge3 quant env (torch 2.12.0+cu130, safetensors 0.7.0, llmcompressor 0.11.0).
• —Source repo: letsrtfm/2x-3090-GA102-300-A1-sglang-inference — see scripts/quantize/quantize_nemotron3_nano_omni.py, scripts/quantize/convert_moe_ct_to_awq.py, patches/052-*, patches/053-*, benchmarks/nemotron3-omni-awq-serve-2026-06-16.md, benchmarks/nemotron3-omni-v0513-caps.json.

Changelog

• —v3 (2026-06-16, late evening) — config + docs only; weights byte-identical to v2. Two things landed:
• —`config.json`: populated quantization_config.modules_to_not_convert and .ignore with the 14-entry exclusion list verified by serving (["sound_encoder","sound_projection","mlp1","vision_model","in_proj","out_proj","conv1d","q_proj","k_proj","v_proj","o_proj","gate","lm_head","embeddings"]). v1 and v2 both shipped these fields empty, which caused SGLang's AWQ loader to attempt to wrap the BF16 Mamba2 in_proj and crash on the Marlin shape constraint. The calibration script never wrote its ignore list to the output config; fixed in the upstream script too so future ships emit it automatically.
• —README: corrected the calibration recipe section (was missing 4 of 8 slices and incorrectly implied multimodal bytes were processed during cal — they're dropped via drop_images=True); replaced the "SGLang preset pending" + "6-modality validate deferred" placeholders with the actual verified serving recipe, 6/6 caps result, and 256K decode benchmark.
• —v2 (2026-06-16, evening) — stripped 5934 stale .weight_zero_point int8 tensors that leaked through from compressed-tensors → AWQ conversion (one per quantized Linear). The AWQ format already encodes the symmetric zero-point in .qzeros; the leftover .weight_zero_point is dead weight at best and broke SGLang's MoE loader at worst — it tried to map the residue onto an experts.w2_weight_zero_point parameter that doesn't exist on the AWQ Linear, raising KeyError on model load (caught after 5 SGLang load attempts across the kernel/TP matrix). convert_moe_ct_to_awq.py upstream is patched so future ships are clean by construction.
• —v1 (2026-06-16, afternoon) — initial AWQ ship from Phase 2 calibration. Inadvertently included the .weight_zero_point residue (see v2) and shipped empty modules_to_not_convert (see v3).