prithivMLmods/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16-GGUF

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16-GGUF

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 is the full-precision reference release of NVIDIA's Nemotron 3.5 Lightning model, a hybrid Mixture-of-Experts architecture interleaving Mamba-2 and MoE layers with select Attention layers, totaling 30B parameters with only 3B active at inference and supporting up to 1M tokens of context (256K on a single H100). Pre-trained on over 20 trillion tokens using an NVFP4 recipe and enhanced with Multi-Token Prediction (MTP) layers for richer training signals, it underwent a four-stage pipeline — pretraining, MTP continued pretraining, supervised fine-tuning on code/math/science/tool-calling data, and multi-environment GRPO reinforcement learning across math, code, instruction-following, and structured-output tasks — supporting configurable reasoning mode (enable_thinking), English plus five additional languages, and speculative decoding via DSpark, DFlash, or MTP for faster generation. This BF16 checkpoint is intended primarily as a starting point for customization — post-training (SFT/RL/distillation), domain adaptation, or producing quantized (NVFP4, W4A16, GGUF) variants — rather than direct production deployment, for which NVIDIA recommends the companion NVFP4 release; on NVIDIA's internal benchmark harness it trails larger sibling Qwen3.6-35B-A3B and Nemotron 3 Super on most agentic coding and reasoning tasks (e.g., 51.56 on SWE-bench Verified, 75.44 on GPQA Diamond) while remaining competitive with Gemma4-26B-A4B and GPT-OSS-20B, deployable on single or multi-GPU H100/H200/GB200/A100 setups via vLLM, and released under the OpenMDW-1.1 license alongside NVIDIA's full pre-training and post-training dataset lineage.

Model Files

Quick Start with llama.cpp

FROM ghcr.io/ggml-org/llama.cpp:full

WORKDIR /app

RUN apt update && apt install -y python3-pip
RUN pip install -U huggingface_hub --break-system-packages

RUN python3 -c 'from huggingface_hub import hf_hub_download; \
    repo="prithivMLmods/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16-GGUF"; \
    hf_hub_download(repo_id=repo, filename="NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.Q4_K_M.gguf", local_dir="/app")'

CMD ["--server", \
     "-m", "/app/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.Q4_K_M.gguf", \
     "--host", "0.0.0.0", \
     "--port", "7860", \
     "-t", "2", \
     "--cache-type-k", "q8_0", \
     "--cache-type-v", "iq4_nl", \
     "-c", "128000", \
     "-n", "38912"]

llama.cpp

LLM inference in C/C++ — https://github.com/ggml-org/llama.cpp