Modelpublic

EryriLabs/Lightning-Herald-30B-A3B-GGUF

sourceHugging Faceotherupdated 1mo agoView on Hugging Face

1likes372downloads

Model Card

Lightning-Herald-30B-A3B GGUF

GGUF builds of EryriLabs/Lightning-Herald-30B-A3B, my Hermes-Agent tune of NVIDIA Nemotron 3.5 Lightning (30B total, 3B active). The full model card, benchmarks and training details live on the main repo. This one is just the files.

Files

Quant	Size	Notes
IQ4_XS	19 GB	The exact file used in the before/after benchmark. Best fit for one 24GB card with a big context.
Q4KM	25 GB	Standard 4-bit. Needs two GPUs or some CPU offload on 24GB.
Q5KM	27 GB
Q6_K	35 GB
Q8_0	35 GB	Close to lossless.

All made with llama.cpp b10413 (convert_hf_to_gguf.py then llama-quantize) from the released BF16 weights. You need a llama.cpp build with Nemotron-H support, so b10413 or newer.

Run

bash

llama-server -m Lightning-Herald-30B-A3B-IQ4_XS.gguf --jinja -ngl 999 -c 32768 --temp 1.0 --top-p 0.95

--jinja matters: it makes llama.cpp render the chat template with your tools and parse tool calls into structured tool_calls. The sampling above is NVIDIA's recommendation for the base model and what the benchmark used.

Point Hermes Agent at http://127.0.0.1:8080/v1. A config preset is on the main repo.

Licence

OpenMDW-1.1, inherited from the base. Training data was Apache 2.0.

Dwain Barnes, EryriLabs