nota-ai/Nemotron-3.5-Lightning-30B-A3B-NVFP4-Global-Pruned-15

Nemotron-3.5-Lightning-30B-A3B-NVFP4-Global-Pruned-15

On top of NVFP4 quantization, this variant applies additional expert pruning. Rather than removing the same number of experts from every block, Nota AI's proprietary global importance-estimation technique measures expert importance across the whole network and keeps only the most important experts on a per-layer basis (global pruning). The number of routed experts therefore varies from layer to layer, recovering accuracy that uniform pruning gives up, while freeing 39% more KV cache on a single 32 GB GPU.

Method clarification. This model does not use uniform expert pruning, in which the same number or proportion of experts is removed from every layer. Instead, we use our proprietary global-scale expert importance score to compare experts across the entire network and determine a different number of retained experts for each layer. Consequently, the pruned architecture has layer-wise variable expert counts, which are explicitly recorded in the model configuration and require the patched model definition provided in this repository.

Architecture

Capacity is concentrated at the two ends of the stack: the first and last six MoE layers keep 116 experts each, and the eleven middle layers carry the whole reduction at 96–100.

Only MoE experts are pruned. The 6 attention layers, 23 Mamba-2 layers, shared experts and the MTP block are untouched, so the 1M-token context window is unchanged.

Every layer's expert count is a multiple of 4, so tensor/expert parallel sizes of 2 and 4 divide evenly. Counts live in config.json as n_routed_experts_per_layer; n_routed_experts stays at 128 — the largest count in the model, held by the unpruned MTP block — so weight mapping still covers every expert.

Quantization is inherited from the base checkpoint and not modified: W4A16_NVFP4 with group_size=16 for the experts (4-bit weights, bf16 activations) and FP8 for a small set of Mamba projections, in NVIDIA ModelOpt format.

Performance

Accuracy

Greedy decoding (temperature 0), context length 139,264 tokens, generation budget max_tokens 131,072, full benchmark sets.

Greedy decoding is used deliberately: it removes sampling variance, so the degradation introduced by compression is measured reproducibly rather than blurred by run-to-run noise.

Per-layer allocation lifts the average by +1.58 points over REAP and leads on 4 of the 5 benchmarks; HumanEval+ ends up above the unpruned NVFP4 checkpoint.

Note. REAP is the state of the art for MoE expert pruning and the method behind most pruned MoE checkpoints published on the Hugging Face Hub. The column above is a REAP baseline built from the same saliency scores and the same 15.6% removal rate as ours, removed uniformly from every layer — so the only difference is how the budget is spread across layers.

Memory

Measured with vLLM 0.27.1, enforce_eager, serving the full 1M-token context window.

Both checkpoints start on a single 32 GB card. Pruning does not change whether the model fits — it changes how much of the card is left for serving.

Two things worth knowing:

• —Pruning reduces memory, not FLOPs. Top-6 routing still activates six experts per token no matter how many remain, so per-request decode throughput is essentially unchanged; the gain is concurrency and footprint.
• —--kv-cache-dtype fp8 does not enlarge the KV pool here. vLLM pads the attention page up to the Mamba state page (attention block size 4176 tokens to ensure that attention page size is >= mamba page size), so the Mamba side sets the floor. At an identical budget, auto and fp8 both give 10.28 GiB / 3,522,560 tokens / 3.36× concurrency on this checkpoint.

Quick Start

export MODEL_CKPT=nota-ai/Nemotron-3.5-Lightning-30B-A3B-NVFP4-Global-Pruned-15
export VLLM_PATH=/path/to/vllm

1. Copy the patched config file

vLLM already calls a get_nemotron_h_config_for_layer hook for models with per-layer expert counts, but the transformers config class that must answer that hook does not implement it. Copy one file over the installed one:

cp patch/configuration_nemotron_h.py \
   $($VLLM_PATH/bin/python -c "import transformers.models.nemotron_h.configuration_nemotron_h as m; print(m.__file__)")

Without it, loading fails with AssertionError: Attempted to load weight (torch.Size([116])) into parameter (torch.Size([128])), because every MoE layer would be built with the maximum expert count instead of its own. Re-apply after any pip install -U transformers.

2. Serve

The deployment recipes from the base model card carry over unchanged — pruning removes experts but leaves the architecture, tokenizer, chat template and MTP block intact. vLLM v0.27.1 or newer.

Max throughput, single GPU:

vllm serve --model $MODEL_CKPT \
    --max-num-seqs 256 \
    --max-num-batched-tokens 16384 \
    --enable-prefix-caching \
    --async-scheduling \
    --mamba-backend flashinfer \
    --mamba-cache-mode align \
    --reasoning-parser nemotron_v3 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice

Interactive, low concurrency, with the DSpark drafter for speculative decoding (export DSPARK_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark):

vllm serve --model $MODEL_CKPT \
    --max-num-seqs 128 \
    --enable-prefix-caching \
    --async-scheduling \
    --speculative_config.model $DSPARK_CKPT \
    --speculative_config.num_speculative_tokens 3 \
    --mamba-backend flashinfer \
    --reasoning-parser nemotron_v3 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice

Multi-GPU (TP4 with expert parallelism):

vllm serve --model $MODEL_CKPT \
    --tensor-parallel-size 4 \
    --enable-expert-parallel \
    --enable-prefix-caching \
    --async-scheduling \
    --mamba-backend flashinfer \
    --mamba-cache-mode align \
    --reasoning-parser nemotron_v3 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice

Expert parallelism is limited to 2 or 4 ranks. --enable-expert-parallel splits each layer's experts across ranks, and this model's per-layer counts are 96, 100 and 116 — divisible by 4 but not by 8. Tensor parallelism without --enable-expert-parallel shards inside each expert instead and is unaffected, so TP8 remains available that way.

Context length. These snippets serve the full 1M-token context window by default. If you are memory-constrained, or want more KV-cache headroom at high concurrency, lower --max-model-len to match your workload.

Only plain single-GPU serving was re-validated on this checkpoint; the speculative-decoding, multi-GPU and W4A16/Ampere paths are inherited from the base model card and were not re-tested here.

Limitations

• —Requires the bundled config file.
• —Expert importance was estimated on English calibration data covering instruction-following and reasoning/agentic/code traces. Other languages may be affected more than the headline numbers suggest.
• —The MTP block was not scored and is left at 128 experts.