WhiskyAKM/Nemotron-3.5-Lightning-30B-A3B-GGUF
NVIDIA-Nemotron-3.5-Lightning-30B-A3B - GGUF
This repository contains GGUF format model files for NVIDIA's NVIDIA-Nemotron-3.5-Lightning-30B-A3B.
These files were converted and quantized using llama.cpp.
Available Files
Model Summary
Model Overview
Model Developer: NVIDIA Corporation
Model Dates: December 2025 - May 2026
Data Freshness:
- The pre-training data has a cutoff date of September 2025.
- The post-training data has a cutoff date of May 2026.
What is Nemotron?
NVIDIA Nemotron™ is a family of open models with open weights, training data, and recipes, delivering leading efficiency and accuracy for building specialized AI agents.
Description
NVIDIA-Nemotron-3.5-Lightning-30B-A3B is a large language model (LLM) trained by NVIDIA.
The model employs a hybrid Mixture-of-Experts architecture, utilizing interleaved Mamba-2 and MoE layers, along with select Attention layers. The Lightning 3.5 model is released alongside speculative decoding methods (DSpark, DFlash, MTP) for faster text generation. The model has 3B active parameters and 30B parameters in total.
This model is ready for commercial use under the OpenMDW-1.1 license.
Usage with llama.cpp
CLI / llama-cli
Reasoning ON (default):
llama-cli \
-m nemotron-3.5-lightning-30b-a3b-Q4_K_M.gguf \
--jinja \
--chat-template-file chat_template.jinja \
-p "Write a Python function to compute Fibonacci numbers." \
--temp 1.0 --top-p 0.95 \
-ngl 99llama-server
Start the OpenAI-compatible server:
llama-server \
-m nemotron-3.5-lightning-30b-a3b-Q4_K_M.gguf \
--temp 1.0 --top-p 0.95 \
-np 1 \
-c 40960 \
--port 8000 \
-ngl 99 \
-fa on \
--jinja \
--chat-template-file chat_template.jinja \
--no-webui \
--fit offWith DFlash Speculative Decoding
Accelerate token generation using the DFlash draft model:
llama-server \
-m nemotron-3.5-lightning-30b-a3b-Q4_K_M.gguf \
-md nemotron-3.5-lightning-30b-a3b-dflash.gguf \
--draft-max 6 \
--temp 1.0 --top-p 0.95 \
-np 1 \
-c 40960 \
--port 8000 \
-ngl 99 \
-ngld 99 \
-fa on \
--jinja \
--chat-template-file chat_template.jinja \
--no-webui \
--fit offAPI Client Example (OpenAI SDK)
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
# Reasoning ON (default)
response = client.chat.completions.create(
model="nemotron-3.5-lightning-30b-a3b",
messages=[{"role": "user", "content": "Write a haiku about GPUs"}],
max_tokens=4096,
temperature=1.0,
top_p=0.95,
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(response.choices[0].message.content)
# Reasoning OFF (direct answer)
response = client.chat.completions.create(
model="nemotron-3.5-lightning-30b-a3b",
messages=[{"role": "user", "content": "What is the capital of Japan?"}],
max_tokens=128,
temperature=1.0,
top_p=0.95,
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(response.choices[0].message.content)Benchmarks
Reasoning Benchmark Evaluations
License and Terms of Use
Governing Download Terms: Use of this model is governed by the OpenMDW-1.1 model license.