sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B

<div align="center" style="background-color:#fff8e1;border:1px solid #f9a825;border-radius:8px;padding:14px 18px;margin-bottom:18px;"> <p><b>⚠️ EXPERIMENTAL / ALPHA — NOT FULLY TESTED</b></p> <p style="font-size:0.95em;">Early-stage derivative, not a fully benchmarked release. Intended as a <b>base for task-specific LoRA fine-tuning</b> (16GB-VRAM GPU deployments, 24GB+ laptops/desktops). Expect rough edges — validate on your own workload before relying on it.</p> </div>

Nemotron 3.5 Lightning 30B-A3B — REAP-20B (general calibration)

A 20B-total / 3B-active derivative of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, pruned with REAP expert pruning (128 → 77 routed experts per MoE layer, 40% sparsity), calibrated on a general-purpose corpus with a slight coding focus (general instruction 44% / coding 33% / math 22%, 538 samples generated by the full model). MTP head stripped. No LoRA included — this is the pre-LoRA base; a domain LoRA (subagent orchestration + coding/tool-calling mix) is the intended next step.

This repository ships:

• —the REAPed model (weights + config, model-*.safetensors)
• —the methodology + recalibration kit (reap/): how the pruning was calibrated, the prompt-bank mix, and how to redo it on your own data.

Why prune

Nemotron 3.5 Lightning is a hybrid Mamba-2 + MoE + Attention model with hidden=2688, moe_intermediate=1856. Neither is divisible by 256, so llama.cpp K/IQ quant formats (256-element superblocks) cannot represent the expert weights — every GGUF quant falls back to block-32 formats (~4.5 bpw floor) regardless of requested bit width. At 30B that floor is ~18GB; at 20B it is ~11.5GB. REAP removes the redundant experts so the model fits smaller quantization budgets without touching the active computation:

• —~37% smaller total memory at the same speed (3B active unchanged)
• —KV cache stays ~6KB/token (only 6 attention layers, 2 KV heads), so long contexts stay cheap on consumer hardware
• —A Q4_0/IQ4_NL GGUF ≈ 11.5GB fits a 24GB MacBook with room for 32K+ context (the block-32 floor is why IQ3-class GGUF quants are not achievable for this architecture at any size)

Calibration data (the important part)

REAP prunes the experts your calibration data doesn't use. This model was calibrated on a general-purpose mix so the surviving experts serve general use:

Completions were generated by the full model (NVFP4, natural thinking, per-domain token budgets). If your workload is heavily domain-specific, re-calibrate on your own data — see `reap/`.

Using this repo

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B",
    torch_dtype=torch.bfloat16, trust_remote_code=True)
tok = AutoTokenizer.from_pretrained("sleepyeldrazi/Nemotron-3.5-Lightning-30B-A3B-REAP-20B")

For llama.cpp / LM Studio on Apple Silicon, quantize to IQ4_NL (~11.5GB):

llama-quantize model.gguf out-IQ4_NL.gguf IQ4_NL
llama-server -m out-IQ4_NL.gguf -c 131072 --reasoning-parser nemotron_v3

Context length: -c 131072 (128K) is a safe default for 16GB-VRAM GPUs and 24GB Macs — KV is only ~6KB/token (6 attention layers, 2 KV heads), so 128K context costs roughly 0.8GB. The model itself supports up to 1M tokens (256K is NVIDIA's single-GPU deployment length); on laptops the practical limit at longer contexts is prefill time, not memory — raise -c (e.g. 262144) if your workload needs it.

Validation (limited — alpha)

Sanity-checked with greedy generation on general QA, coding, and math prompts: coherent analysis, correct answers (e.g. math arithmetic verified), no repetition/looping. Full benchmark suites were not run. The closest published pruning analogue (Qwen3-30B-A3B, 128 experts, 8 active) retained ~99.8% of baseline at 50% sparsity.

Sources & how to make one yourself

• —Base model: NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
• —Algorithm: REAP — Router-weighted Expert Activation Pruning (Cerebras; implemented in vllm-project/llm-compressor)
• —Pruning tooling: llm-compressor's REAPPruningModifier, plus 3 small patches for the NemotronH architecture (documented in `reap/README.md`)
• —Calibration data: 538 samples generated by the full model from general instruction, coding, and math prompt banks (openhermes-style instruction data, DeepSeek coding-trajectory prompts, metamath-style word problems) — see `reap/generate_general_data.py`
• —Synthetic-data guidance: NVIDIA's Build a Bash Agent with Synthetic Data and sft-data-blending patterns
• —Why 20B / the quant floor: this architecture's hidden=2688 / moe_intermediate=1856 are not 256-divisible, so llama.cpp GGUF quants floor at block-32 formats (~4.5 bpw); at 20B that is ~11.5GB instead of ~18GB

To reproduce on your own data: serve the full model, generate your domain corpus with reap/generate_general_data.py (or your own loader), then run reap/reap_nemotron.py --data yours.jsonl --sparsity 0.40. Full step-by-step, patches, and hardware notes: `reap/README.md`.

Related

• —LoRA adapter + IQ4_NL GGUF (recovery pass + quantized for laptops): Nemotron-3.5-Lightning-30B-A3B-REAP-20B-LoRA-IQ4NL
• —Base: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

License

OpenMDW License Agreement, version 1.1 (same as the base model). See LICENSE. If you upload derived model files, carry the same license and attribute NVIDIA for the base weights.