impacte/NVIDIA-Nemotron-Nano-9B-v2-GGUF

NVIDIA-Nemotron-Nano-9B-v2 (GGUF)

<a href="https://console.runpod.io/hub/template/m7g1jafoep?ref=97dt77nr"> <img src="https://img.shields.io/badge/Deploy%20on%20RunPod-16GB%20GPU-8A2BE2" alt="Deploy on RunPod"> </a>

    

GGUF conversions of `nvidia/NVIDIA-Nemotron-Nano-9B-v2`, a general-purpose instruction-tuned reasoning model from NVIDIA's Nemotron family, converted for use with llama.cpp.

Two quantizations are provided:

Model Summary

Files

impacte/NVIDIA-Nemotron-Nano-9B-v2-GGUF/
├── NVIDIA-Nemotron-Nano-9B-v2-bf16.gguf     # bf16 GGUF (17.79 GB)
├── NVIDIA-Nemotron-Nano-9B-v2-Q4_K_M.gguf   # Q4_K_M GGUF (6.53 GB)
└── .gitattributes                            # LFS tracking

Usage

llama.cpp (local inference)

# bf16 (full quality, 24 GB+ VRAM)
llama-server \
  -m NVIDIA-Nemotron-Nano-9B-v2-bf16.gguf \
  --ctx-size 8192 \
  --port 8080

# Q4_K_M (single 16 GB GPU)
llama-server \
  -m NVIDIA-Nemotron-Nano-9B-v2-Q4_K_M.gguf \
  --ctx-size 32768 \
  --port 8080

Then call the OpenAI-compatible endpoint:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "NVIDIA-Nemotron-Nano-9B-v2-Q4_K_M",
    "messages": [
      {"role": "user", "content": "Explain what a Tauri v2 app is."}
    ]
  }'

llama-cpp-python

from llama_cpp import Llama

llm = Llama(
    model_path="NVIDIA-Nemotron-Nano-9B-v2-Q4_K_M.gguf",
    n_ctx=32768,
    n_gpu_layers=-1,  # offload all layers to GPU
)

About the base model

NVIDIA-Nemotron-Nano-9B-v2 is a compact, instruction-tuned reasoning model in NVIDIA's Nemotron family, trained on NVIDIA's post-training and pretraining datasets. It supports multiple languages (en, es, fr, de, it, ja) and is designed for efficient local deployment with a hybrid Mamba-2 + attention architecture.

License & Attribution

• —Base model: nvidia/NVIDIA-Nemotron-Nano-9B-v2 — released under the NVIDIA Open Model License. Review before commercial use.
• —GGUF conversion: performed with llama.cpp's convert_hf_to_gguf.py (bf16) and llama-quantize (Q4KM).

Note: This is a GGUF conversion of a model under the NVIDIA Open Model License. Ensure your use complies with that license.

Limitations

• —The bf16 GGUF (17.79 GB) preserves full precision but is large — needs 24 GB+ VRAM.
• —The Q4_K_M GGUF (6.53 GB) fits on a single 16 GB GPU but trades some precision for size.
• —The base model's general capabilities are retained; this is a direct conversion with no additional fine-tuning.