Modelpublic

pablohassan/Qwen3.6-27B-NVFP4-FR

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

1likes75downloads

Model Card

Qwen3.6-27B-NVFP4-FR

An NVFP4 (W4A4) quantization of Qwen/Qwen3.6-27B, calibrated on a French on-policy corpus, with a per-layer precision map determined experimentally to preserve French writing quality at a level statistically indistinguable from the FP8 reference.

Built for inference on NVIDIA DGX Spark (GB10, sm_121) under vLLM, paired with the `Qwen3.6-27B-DSpark-FR` drafter for speculative decoding.

French-focused, but broadly useful: the methodology (domain-specific calibration + per-layer precision map + blind quality certification) applies to any language or domain.

What this model brings

Most public NVFP4 quants quantize all linear layers and report out-of-context speed numbers with no quality protocol. This one instead documents:

1.A per-layer precision map from empirical exploration — the 48 Gated DeltaNet layers are kept in BF16 because quantizing them measurably degrades quality.
2.Domain-specific calibration (French on-policy).
3.Quality certification via token-level ΔNLL (with self-reference-bias control) + a blind A/B test judged by a panel.

Precision map

Component	Precision
Attention q/k/v/o, MLP gate/up/down	NVFP4 W4A4
Gated DeltaNet layers (`linear_attn`)	BF16 (protected)
`lm_head`, `embed_tokens`, vision tower	BF16
KV cache	auto (BF16)

Rationale: quantization error in the decay gates of the recurrent layers compounds multiplicatively through the state, degrading coherence. Three more aggressive variants (GDN in FP8) were tested and rejected at the quality gates (see the code repo report).

Performance (measured, DGX Spark GB10, vLLM, with DSpark-FR drafter n=8)

Metric	Value	Conditions
Throughput @ c1 (held-out FR, 300 tok, temp 0.6)	18.56 tok/s	single GPU, single request
Throughput @ c1, TP=2 (2× DGX Spark)	28.79 tok/s	×1.55, drafter intact
Throughput @ c1 (long FR editorial, 800 tok, temp 0)	~14–15 tok/s	single GPU
Aggregate throughput @ c16	~54 tok/s	16 requests
No-speculation floor	8.2 tok/s	reference

Honest numbers: measured on a single GB10 (273 GB/s memory bandwidth). Decode is bandwidth-bound, so these are close to the hardware optimum. Higher numbers reported elsewhere come from multi-GPU (aggregate bandwidth) or a quality-degraded body.

Multi-node: the model shards in tensor parallel across two GB10s (QSFP, 1.9 µs RDMA) at 28.79 tok/s, with 35.65 GiB free KV per worker at 32k. At 32k the KV cache is 2.0 GiB/sequence (4× smaller than a dense equivalent: 48/64 layers are fixed-state Gated DeltaNet). KV stays BF16 with the drafter (fp8 KV + non-causal attention is blocked on sm_121 — vllm#41559).

Quality (certified vs FP8 reference)

—Token-level ΔNLL (150 held-out FR prompts, symmetry control): +0.016 nats ≤ 0.02 threshold → PASS.
—Perplexity ratio: ≤ 1.03 → PASS.
—Blind A/B (50 pairs, 1 operator + 3 independent blind judges): FP8 win rate 52.8% (p = 0.43) → statistically indistinguable.

Tool-calling — corrected note (2026-07-14). An earlier run measured weak tool-calling (~44% on tool-eval-bench) and this model was described as chat-focused. That was not the model: it was a vLLM bug (#43559) where prefix caching combined with speculative decoding corrupts the recurrent Gated-DeltaNet state on hybrid bodies, silently degrading tool-calling. With the fix (vLLM PR #48361) — or simply with prefix caching disabled — this body scores 88–91/100 on tool-eval-bench (Parameter Precision and Multi-Step Chains: 100%). It is suitable for tool-heavy agentic workflows once served correctly. See the drafter card caveat.

Usage (vLLM)

Requires a recent vLLM main build (DSpark drafter support) with VLLM_USE_DEEP_GEMM=0, VLLM_TEST_FORCE_FP8_MARLIN=1.

bash

vllm serve pablohassan/Qwen3.6-27B-NVFP4-FR \
  --quantization compressed-tensors --kv-cache-dtype auto \
  --max-model-len 32768 --gpu-memory-utilization 0.60 \
  --speculative-config '{"method":"dspark","model":"pablohassan/Qwen3.6-27B-DSpark-FR","num_speculative_tokens":8}'

Without the drafter it runs as standard decode (~8 tok/s floor).

Lineage & license

—Base: Qwen/Qwen3.6-27B (Apache 2.0).
—Quantization: llm-compressor (compressed-tensors, nvfp4-pack-quantized), 256 calibration samples (192 FR on-policy + 64 EN).
—License: Apache 2.0 (inherited).