joshebbs/qwen3.8-27b-uncensored-nvfp4-modelopt

Qwen3.8-27B-Uncensored-NVFP4 (ModelOpt)

NVFP4 quantization of JonathanColetti/Qwen3.8-27B-Uncensored, produced with NVIDIA TensorRT Model Optimizer 0.43.0 for Blackwell-class inference under vLLM.

~65 GB bf16 → 19.2 GiB. The multi-token-prediction head and the vision tower are both retained.

What was quantized

400 linear layers to NVFP4 (block size 16, FP8 scales):

Qwen3.8-27B is a hybrid stack — 64 layers of 3× (Gated DeltaNet → FFN) + 1× (Gated Attention → FFN). The DeltaNet decay and beta projections (in_proj_a / in_proj_b) are low-rank and precision-sensitive, so they are left at bf16 along with the causal conv1d.

Fused-layer constraint (important if you re-roll this yourself)

vLLM does not instantiate the DeltaNet input projections separately. It fuses in_proj_qkv + in_proj_z into a single MergedColumnParallelLinear named in_proj_qkvz, and in_proj_b + in_proj_a into in_proj_ba. Every shard of a fused layer must share one precision, or loading aborts during model construction — before a single weight is read:

ValueError: Detected some but not all shards of
language_model.model.layers.0.linear_attn.in_proj_qkvz are quantized.
All shards of fused layers to have the same precision.

So in_proj_z must be quantized together with in_proj_qkv, even though it is a gate. Excluding in_proj_a and in_proj_b is fine because they are excluded together, which leaves in_proj_ba uniform. The same rule applies to qkv_proj and gate_up_proj.

This checkpoint has been verified to satisfy that constraint: every fused group is internally single-precision, checked per parent module.

exclude_modules naming

exclude_modules is matched against vLLM's module prefixes, by exact string equality first. Recent transformers emits the checkpoint hierarchy as model.language_model.…, whereas vLLM builds language_model.model.… — so a config exported verbatim will silently fail to match, and vLLM will try to quantize layers that have no scales. The exclusion list here is written in both conventions, and was validated by running vLLM's own is_layer_skipped / is_layer_excluded over every module in the checkpoint and confirming its decision matches whether that module actually carries weight_scale tensors.

Calibration

256 samples from `garage-bAInd/Open-Platypus`, batch size 16, max sequence length 1024, max calibration. No fine-tuning, no additional training data.

Serving

vLLM on Blackwell:

vllm serve joshebbs/qwen3.8-27b-uncensored-nvfp4-modelopt \
  --quantization modelopt_fp4 \
  --kv-cache-dtype fp8 \
  --attention-backend flashinfer

The chat template opens a <think> block by default; pass enable_thinking=False to apply_chat_template for direct answers. Qwen's recommended sampling is temperature=1.0, top_p=0.95, top_k=20.

Note that transformers cannot load this checkpoint directly — NVFP4 packs two 4-bit values per byte, so weights are stored at half width and a plain from_pretrained will report shape mismatches. Use a runtime that understands modelopt_fp4.

Tool calling

The chat template emits tool calls in Qwen's XML dialect (<tool_call><function=name><parameter=k>v</parameter></function></tool_call>), so vLLM needs the matching parser. Without both flags, any client sending tool_choice: "auto" gets 400 "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set:

--enable-auto-tool-choice --tool-call-parser qwen3_coder

Reasoning parser + max_tokens

With --reasoning-parser qwen3, a response truncated inside the <think> block (finish_reason: "length") comes back with both content and reasoning_content empty — the parser needs the closing </think> before it will emit anything. This looks alarmingly like a corrupted checkpoint but is purely a budget artifact. Either give thinking mode enough headroom (2500 tokens was still not always enough for a verbose "explain in detail" prompt) or set enable_thinking=False.

Deployment: 2× NVIDIA DGX Spark (GB10), TP=2

Verified serving on a pair of DGX Spark GB10 nodes joined by a direct 200 Gb/s QSFP link, tensor-parallel across the two, one GPU per node:

vllm serve joshebbs/qwen3.8-27b-uncensored-nvfp4-modelopt \
  --served-model-name qwen38-27b \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 131072 \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.75 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --trust-remote-code \
  --dtype auto \
  --kv-cache-dtype fp8 \
  --quantization modelopt_fp4 \
  --attention-backend flashinfer \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  -tp 2 --nnodes 2 --node-rank 0 \
  --master-addr 10.10.10.1 --master-port 29501

Rank 1 runs the same line with --node-rank 1.

Measured startup (warm compile cache)

First run on a cold FlashInfer cache is far slower — budget 25–30 min. Raise the NCCL store timeout (300 s default is not enough for multi-node JIT) or rank 1 will drop out mid-compile while rank 0 is still building kernels.

Measured throughput (TP=2, live server)

vllm bench serve, --dataset-name random --random-input-len 512 --random-output-len 256 --ignore-eos, hitting the running OpenAI chat endpoint. Warm cache, negligible background load.

Single-user latency is fine (~192 ms first token, ~48 ms per subsequent token). Under 8-way concurrency total-token throughput scales 3.6×; TTFT balloons because prefills queue against --max-num-seqs 4 / --max-num-batched-tokens 8192, but decode ITL barely moves — the ceiling is batch admission, not compute. Raise --max-num-seqs if you need lower TTFT under bursts.

Reproduce:

docker exec -e HF_HUB_OFFLINE=1 vllm_node vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 --endpoint /v1/chat/completions \
  --model qwen38-27b \
  --tokenizer /root/.cache/huggingface/hub/models--joshebbs--qwen3.8-27b-uncensored-nvfp4-modelopt/snapshots/<hash> \
  --dataset-name random --random-input-len 512 --random-output-len 256 \
  --num-prompts 32 --max-concurrency 8 --ignore-eos

Notes specific to this stack

• —Both nodes must hold the same checkpoint revision. vLLM resolves the repo id to a local snapshot path per node; if one node's HF cache is a revision behind, each rank silently loads different weights and the run hangs in distributed init rather than reporting a mismatch. Check refs/main on both.
• —Prefix caching puts the Mamba/DeltaNet cache in align mode, which vLLM flags as experimental for this architecture. Drop --enable-prefix-caching first if you see output corruption.
• —The MTP head ships in the checkpoint but is not loaded unless you configure speculative decoding; speculative_config=None leaves model-mtp-grafted.safetensors unused.

Refusal behaviour — inherited, not re-measured

The base checkpoint's author reports 12/100 refusals vs 98/100 for stock Qwen3.8-27B on the test split of `mlabonne/harmful_behaviors`, measured in non-thinking mode, using Heretic (200-trial search co-minimizing refusal count against KL divergence from base).

Those numbers describe the bf16 source, not this quantization. The refusal edit lives in o_proj and down_proj, which are exactly the tensors compressed here, so the effect could in principle be attenuated. The bf16 source was spot-checked as compliant before quantization.

Refusals are reduced, not eliminated, in the source model.

Credits

• —Qwen/Qwen3.8-27B — base model
• —JonathanColetti/Qwen3.8-27B-Uncensored — Heretic abliteration and MTP graft
• —Heretic — abliteration method
• —NVIDIA TensorRT Model Optimizer — NVFP4 quantization