Modelpublic

primitive-ai/Nex-N2.5-mini-NVFP4

sourceHugging Faceapache-2.0updated 15d agoView on Hugging Face

0likes1.5kdownloads

Model Card

<h3 align="center">A third of the size, 1.5× the throughput, and the eval cannot tell it from BF16.</h3>

NVFP4 quantization of <a href="https://huggingface.co/nex-agi/Nex-N2.5-mini">nex-agi/Nex-N2.5-mini</a> to 23.91 GB, on one 96 GB card. All 40 layers of routed experts at 4 bits; attention, linear attention, shared experts, embeddings and the vision tower stay BF16.

Three builds, one measurement. We quantized this model's experts three ways and measured all of them against the BF16 original under one protocol: this one, an FP8 build at 38.13 GB, and an mixed build at 26.04 GB. On accuracy the four are one band. They separate on size and on speed, and which one is fastest depends on your concurrency. This is the smallest, and the fastest under load.

Why this quant

—🗜️ 23.91 GB against 70.24 GB in BF16. The routed experts are 91.8% of this model's weight, so quantizing them alone gets almost the whole reduction.
—🎯 Nothing measurable is lost. 90.5 on the 1,170-item knowledge suite and 77.0 on the 200-item tool-calling suite, against the BF16 original's 89.6 and 75.3 on the same box. The original is at the bottom of the band, not the top.
—⚡ 1,019 tok/s at concurrency 32, 1.53× the BF16 original's 664. Single-stream it runs 145 tok/s against BF16's 141.
—🧠 A 2.74M-token KV pool at 32K context, three times what BF16 leaves room for on the same card, which is what lets one GPU hold 83 concurrent 32K sessions.
—🧩 Stock vLLM, no patches. compressed-tensors, nvfp4-pack-quantized, served by the released 0.28.0 image.

Serve it

bash

hf download primitive-ai/Nex-N2.5-mini-NVFP4 --local-dir ./Nex-N2.5-mini-NVFP4

docker run --gpus all --ipc=host -p 8000:8000 -v $PWD:/models \
  vllm/vllm-openai:v0.28.0 \
  --model /models/Nex-N2.5-mini-NVFP4 \
  --max-model-len 32768 --gpu-memory-utilization 0.92 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Qwen3_5MoeForConditionalGeneration is in the 0.28.0 release, so no nightly is needed. The numbers on this page were measured on 0.28.1rc1.dev472, and 0.28.0 serves the same weights with the same kernels (FlashInfer CUTLASS for the NVFP4 experts). Boot takes about 210 s on an idle card.

Two things about this model's chat template are worth knowing before you wire it up. It emits an empty thinking block by default, so replies arrive with two leading newlines and the reasoning field holds only whitespace; a short correct answer looks like "\n\n4". And reasoning_effort (low / medium / xhigh) is a per-request field here, unlike some other reasoning models where it is fixed at server start.

Measured

One RTX PRO 6000 Blackwell, 96 GB, one card. The 1,170-item knowledge suite and the 200-item tool-calling suite, temperature 0.6 / top_p 0.95 / top_k 20, thinking on, a 16,384-token budget, concurrency 32, auto-scored with no LLM judge. Throughput is 8K in / 512 out, prefix-cache free, two seeds per cell. Every row below was run on the same box in the same sitting.

build	size	knowledge	tool-calling	call	abstain	finished	tok/s @1	tok/s @32
BF16 original	70.24 GB	89.6	75.3	79.7	58.7	99.7%	141.0	664
FP8 experts	38.13 GB	90.9	75.5	79.4	60.0	99.5%	163.8	901
mixed	26.04 GB	90.6	76.5	80.9	58.7	99.2%	147.2	990
this repo, NVFP4	23.91 GB	90.5	77.0	81.9	57.5	99.3%	145.3	1019

Tool-calling is the mean of two runs per build (three for BF16). Pooled within-build spread is 0.60 standard deviations over those nine runs, and the four build means span 1.7, so the column is one band. Read the ordering with that in mind: it runs opposite to precision, which no mechanism would predict, and is the clearest evidence that this suite cannot resolve the difference between 4-bit and 16-bit experts on this model.

Speed does separate, and it crosses over. FP8 experts decode 13% faster single-stream because the 4-bit path pays for activation quantization and group scales on every token, a cost that does not amortize at batch 1. At concurrency 32 the ranking inverts and NVFP4 wins by 13%, because there the bottleneck is weight bandwidth and 4-bit moves half the bytes. Pick by the concurrency you actually run.

Comparable with our other models

Accuracy numbers move for reasons that have nothing to do with the model: a shorter token budget, a different temperature, or whether the model was allowed to reason at all. So every number in this table, on this card and on our other cards, comes from the one fixed protocol described above, the same 1,370 items, auto-scored, no LLM judge.

model	shape	size	overall	knowledge	call	abstain	finished	out/answer
Laguna-XS-2.1	31 B MoE	19.3 GiB	81.7	83.8	68.4	73.5	98.9%	1097
Nemotron-3.5-Lightning-30B-A3B	30 B MoE+Mamba	19.2 GiB	87.1	87.9	85.4	70.5	97.9%	1429
Ornith-1.5-35B-A3B	35 B MoE	22.6 GiB	88.7	91.7	74.4	60.0	99.3%	760
Muse-Glimmer-30B	30 B MoE	20.4 GiB	86.6	88.8	78.6	54.5	99.7%	800
Qwen3.8-27B	27 B dense	20.7 GiB	88.8	90.4	85.5	54.5	99.7%	651
Granite-4.2-30B	30 B dense	18.1 GB	85.5	86.2	85.8	60.8	98.5%	1502
Nex-N2.5-mini NVFP4 (this repo)	35 B MoE, 3 B active	23.91 GB	88.5	90.5	81.9	57.5	99.3%	524
Nex-N2.5-mini mixed	35 B MoE, 3 B active	26.04 GB	88.5	90.6	80.9	58.7	99.2%	504
Nex-N2.5-mini FP8	35 B MoE, 3 B active	38.13 GB	88.7	90.9	79.4	60.0	99.5%	545
K2-Horizon-MoVA-36B-A4B NVFP4	37 B MoE+MoVA, 4 B active	36.7 GB	84.1	86.5	71.8	60.4	95.6%	1234
K2-Horizon-MoVA-36B-A4B mixed	37 B MoE+MoVA, 4 B active	44.5 GB	84.9	87.3	73.5	58.9	96.3%	1118
Laguna-S-2.1	110 B MoE	64.0 GiB	84.3	87.1	64.6	81.0	97.3%	995
Qwen3.8-Flash-Next	180 B MoE, 6 B active	183.7 GB	90.3	92.2	84.8	56.7	99.5%	686

overall pools the two suites as 1,370 items, weighted 85.4% knowledge and 14.6% tool calling by item count. Read it with `finished`: overall scores an answer that overran the token budget as wrong, and cannot say whether the model needed the room or failed to stop. A gap under 1.0 in overall is a tie. Sizes are as each card reports them, which mixes GB and GiB.

What's quantized to what

Nex-N2.5-mini is 35.11 B parameters, and they are distributed very unevenly:

	params	share
routed experts, 40 layers × 256 × (gate+up+down)	32.21 B	91.8%
linear attention (Gated DeltaNet), 30 layers	1.01 B	2.9%
`embed_tokens` and `lm_head`, untied, vocab 248320	1.02 B	2.9%
vision tower, 27 blocks	0.45 B	1.3%
full attention, 10 layers	0.27 B	0.8%
shared experts, routers, norms	0.15 B	0.4%

tensors	count	format
routed experts on all 40 layers	30,720 modules	NVFP4, group 16
everything else	946 tensors	BF16, byte-identical to the source

compressed-tensors, format nvfp4-pack-quantized, one config group, W4A4 with group-16 weight scales and a per-module global scale. The source ships experts as two batched 3-D tensors per layer; they are unfolded into per-expert modules, which is the layout vLLM's loader consumes. gate and up share one weight_global_scale per expert. vLLM fuses those two halves and keeps a single scale, warning and taking the maximum when they disagree, which silently shrinks half the weights, so the build is audited for that before release: 20,480 fused groups checked, no mismatches.

Weights-only round-to-nearest, no calibration. Nothing in the quantization is fitted to data.

<img src="https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8/resolve/main/assets/primitive-logo.png" alt="Primitive" width="34"> <a href="https://primitive.com">primitive</a> · <a href="https://huggingface.co/primitive-ai">more models</a> · inference economics for production LLM systems