Modelpublic

Ostfralla/Qwen3.8-27B-NVFP4-NInfer

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

27likes4.6kdownloads

Model Card

Qwen3.8-27B — NVFP4 for NInfer

An NVFP4 (4-bit, Blackwell FP4 tensor-core) artifact of Qwen/Qwen3.8-27B for the NInfer engine.

The artifact published by the NInfer author, neroued/Qwen3.8-27B-NInfer, uses the groupwise-int weights profile (Q4/Q5/W8). This one uses nvfp4, which runs on the FP4 tensor cores of Blackwell GPUs.

Requirements

—Blackwell GPU (compute capability 12.0 — RTX 5090 and similar). NVFP4 will not run on Ada or Hopper.
—~17 GiB VRAM for weights; a 32 GB card for useful context.
—NInfer with `qwen38-nvfp4-support.patch` (included in this repo) — the identity qwen3.8-27b + nvfp4 is not registered in upstream NInfer yet, so the engine refuses the artifact with artifact identity 'qwen3.8-27b/nvfp4' is not supported by target 'qwen3_6_27b'. Upstream issue: https://github.com/Neroued/ninfer/issues/25 — if that lands, the patch becomes unnecessary.

Measured results (RTX 5090, 32 GB)

Same engine, same flags, same problems, greedy (temperature 0), 6 concurrent slots.

Quality — indistinguishable from the published artifact

Two independent capability axes, 224 problems, greedy (deterministic, so the comparison is paired):

benchmark	this artifact (nvfp4)	neroued (groupwise-int)	disagreements
HumanEval+ (code, 164)	152/164 = 92.68%	152/164 = 92.68%	3 vs 3
AIME25 + AIME26 (math, 60)	55/60 = 91.67%	55/60 = 91.67%	5 vs 5
total (224)	207	207	8 vs 8

Identical totals on both axes, with the per-problem disagreements splitting evenly. The individual AIME sets pulled in opposite directions (int +2 on aime25, nvfp4 +2 on aime26) and cancelled — a reminder that a single 30-problem set is not enough to conclude anything.

Speed — 1.56x to 1.98x faster wall clock

benchmark	nvfp4 wall	int wall	speedup	nvfp4 decode	int decode
HumanEval+ (164)	694 s	1,080 s	1.56x	882 tok/s	385 tok/s
AIME25 + 26 (60)	1,287 s	2,544 s	1.98x	807 tok/s	414 tok/s

The AIME figure is the cleaner one: there both artifacts generate comparable token volumes, so the throughput gap shows through undiluted. On HumanEval+ nvfp4 generated 47% more tokens, which drags its wall-clock advantage down even though it is doing more work.

Single-stream decode is 202 tok/s with MTP speculative decoding, and prefill runs at ~5,950 tok/s (vs ~1,700 for llama.cpp Q5KXL).

VRAM

Weights occupy 16.78 GiB loaded with MTP, against 15.92 GiB for the groupwise-int artifact — this build is slightly larger in VRAM, and buys speed rather than memory. Against llama.cpp Q5KXL (18.83 GiB) it is 2 GiB smaller.

Usage

bash

git clone https://github.com/Neroued/ninfer.git && cd ninfer
git apply qwen38-nvfp4-support.patch     # included in this repo

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build --parallel

./build/apps/ninfer-serve qwen3_8_27b_nvfp4.ninfer \
  --host 127.0.0.1 --port 18080 \
  --max-concurrency 6 --max-context 262144 --kv-capacity auto \
  --default-max-tokens 80000 --pending-timeout-ms 900000 \
  --kv-dtype int8 --spec mtp --draft-tokens 4 --lm-head-draft

Pass no sampler flags: NInfer applies Qwen3.8's own presets per thinking mode (thinking 1.0 / 0.95 / 20; non-thinking 0.7 / 0.80 / 20 / presence 1.5). Setting --temperature overrides the preset and disables the per-mode switch.

If the build fails on `char8_t` / `type_traits`

CUDA's nvcc cannot parse GCC 16's libstdc++ headers, so on an up-to-date Arch or CachyOS the configure step dies with /usr/include/c++/16.1.1/type_traits: error: identifier "char8_t" is undefined. Point it at GCC 15:

bash

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-15 \
  -DCMAKE_C_COMPILER=/usr/bin/gcc-15 -DCMAKE_CXX_COMPILER=/usr/bin/g++-15

Verified 2026-08-15 against upstream 604bdc5: clean clone, this patch, this build command, then the artifact below loads and generates correctly.

How it was built

1.llm-compressor NVFP4 over the BF16 original, quantizing the DeltaNet projections (in_proj_qkv, in_proj_z, out_proj). in_proj_a/in_proj_b must stay BF16: N=48 and CUTLASS FP4 requires N % 64 == 0.
2.Reconcile fused-group global scales. NInfer fuses in_proj_qkv with in_proj_z into one object carrying a single weight_global_scale, but llm-compressor quantizes each module independently, so members disagree (measured factors 1.12–1.82 apart) and the converter rejects the source.

Since global_scale = 448*6 / amax is inversely proportional to amax, the scale the concatenated matrix would have produced is simply the smallest member scale:

   G = 448*6 / amax(concat) = 448*6 / max_i(amax_i) = min_i(G_i)

Members are then re-quantized from BF16 under that shared scale rather than rescaled in place, which avoids double-rounding block scales that are already E4M3. Measured cost: none — mean reconstruction error 0.09471 → 0.09470, zero underflowed block scales. 47 groups needed this, all linear_attn.

1.Convert with tools/convert/qwen3_8_27b/convert_nvfp4.py.

Step 2 is tools/convert/qwen3_8_27b/fuse_weight_scales.py in the fork.

Known caveats

—Under greedy decoding this artifact truncated 5 of 164 HumanEval+ problems at a 50k token cap, against 3 for the groupwise-int artifact — it falls into reasoning loops slightly more often. With sampling enabled this does not occur.
—Quality is validated on HumanEval+ and AIME25/26. Not on long-context or multilingual tasks.
—The engine patch is not upstream yet (https://github.com/Neroued/ninfer/issues/25).

License

Apache-2.0, matching both Qwen3.8-27B and NInfer.