Feargal/Nemotron-Orchestrator-8B-AWQ-W4A16

Nemotron-Orchestrator-8B-AWQ-W4A16

4-bit AWQ W4A16 quantization of `nvidia/Nemotron-Orchestrator-8B`, produced with llm-compressor and served with SGLang.

nvidia/Nemotron-Orchestrator-8B is NVIDIA's ToolOrchestra orchestration model (arXiv:2511.21689), itself a GRPO-RL fine-tune of Qwen/Qwen3-8B. It routes a request across tools and expert models, reasoning in a <think> block before answering. This quant exists to cut its VRAM footprint and decode latency when it is co-tenanted on one GPU alongside larger models.

What changed

Only the 252 transformer nn.Linear modules (36 layers x 7) are quantized to 4-bit, group size 128, W4A16_ASYM. lm_head is excluded and stays BF16 — the embeddings are untied on this architecture, so lm_head is a real 151936 x 4096 matrix whose rounding error would land directly on the token logits that carry the routing decision. embed_tokens also stays BF16. That is why the checkpoint is ~6.1 GB and not the ~4.1 GB a naive "8B at 4 bits" estimate suggests.

Calibration used 256 sequences of the model's actual serving prompt shape — its system prompt plus a routing instruction over FEVER claims, through the chat template — rather than generic chat data. Post-training quantization fits scales to observed activation ranges, and this model only ever sees one prompt shape in deployment.

Measured

Both arms served on one H100 through the same launcher and the same probe set, so the comparison is like-for-like rather than across cards.

Known regressions

Read these before deploying it.

• —One routing disagreement in ten, and it is in the unsafe direction. On "Lost is a French drama." the quant routes to direct_reply where BF16 routes to cache_search. direct_reply answers from parametric memory with no retrieval at all, which for a fact-checking pipeline is the worst way for a routing error to go.
• —`<think>` block not observed on the sampled probes, where BF16 emitted one averaging ~160 words. This was measured on a single sample per claim at temperature 0.1 and the generation still produced 341 output tokens, so it is not settled whether the reasoning was lost or merely emitted untagged — untagged reasoning still parses downstream. Treat it as an open question, not a confirmed regression.
• —Difficulty MAE 0.09 is not free if you threshold on that score. Cantivia tiers its debate budget at 0.3 and 0.7, so drift of this size can move a claim a tier.

Serving

python -m sglang.launch_server \
  --model-path Feargal/Nemotron-Orchestrator-8B-AWQ-W4A16 \
  --dtype bfloat16

The compressed-tensors quantization config is read from config.json; no --quantization flag is needed.

Licence

Inherits the NVIDIA License from the base model — a complete copy ships as LICENSE in this repo. Section 3.3 limits the work and every derivative of it, including this quant, to non-commercial use: "non-commercial research activities or non-commercial research publications only". The base model card additionally states it is for research and development only.

Attribution and all notices from the source model are retained per section 3.1. This repository contains no new training; it is a numerical compression of NVIDIA's weights.