Modelpublic

originalorz/Qwen3.6-27B-Claude-Opus-Distilled-NVFP4

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes8downloads

Model Card

Qwen3.6-27B-Claude-Opus-Distilled-NVFP4 (multimodal, vLLM-ready)

NVFP4 quantization of [rico03/Qwen3.6-27B-Claude-Opus-Reasoning-Distilled](https://huggingface.co/rico03/Qwen3.6-27B-Claude-Opus-Reasoning-Distilled), with the vision tower preserved (BF16) and everything wired so it actually loads in vLLM.

To our knowledge this is the first vLLM-loadable NVFP4 build of the Opus-distilled Qwen3.6-27B — the upstream repo only ships BF16 (54 GB) and GGUF. This build is 18.4 GB, fits a single 32 GB Blackwell GPU (RTX 5090 etc.), and keeps image + video understanding working.

Why the distilled model at all?

The distill (14k Claude Opus 4.6 reasoning traces, methodology by Jackrong) fixes the most painful production behaviour of the base reasoning model: burning the entire max_tokens budget on the thinking chain and returning content: null. Our production A/B (same NVFP4 precision, same TP2 setup):

Task	Official Qwen3.6-27B-NVFP4	This model
Chinese copywriting	1671 tok / 15.2s	688 tok / 6.4s
Coding task	❌ burned 2048-tok budget, empty answer	✅ complete, 722 tok
Math word problem	✅ correct, 1624 tok	✅ correct, 352 tok
Image understanding	❌ burned budget, no answer	✅ correct, 138 tok
Long-doc summary	656 tok	241 tok
Decode speed (2×RTX 5090, TP2)	108.7 tok/s	109.3 tok/s (parity)

Same speed, ~1/2–1/3 the tokens, zero empty answers in our suite. Long-context verified to 240K tokens input on 2×32 GB.

Usage (vLLM ≥ 0.24)

bash

vllm serve <this-repo> \
  --tensor-parallel-size 2 \
  --kv-cache-dtype fp8 \
  --max-model-len 262144 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --reasoning-parser qwen3 \
  --trust-remote-code

Single 32 GB GPU also works (reduce --max-model-len, add --enforce-eager if CUDA-graph capture OOMs).

⚠️ Two settings that are NOT optional

1.`--tool-call-parser hermes` — the distilled model emits <tool_call>{...}</tool_call> (hermes format). With qwen3_coder parser tool calls silently degrade to plain text.
2.Don't swap the bundled tokenizer/chat template. This repo intentionally mixes:
3.tokenizer.json / tokenizer_config.json from the official base (defines the vision special tokens — with the upstream distill tokenizer, video inputs crash Qwen3VLProcessor);
4.chat_template.jinja from the distill (drives the hermes tool format — with the official template the model stops emitting tool calls).

Each mismatch breaks exactly one capability; we hit both so you don't have to.

How it was made

—Quantizer: llm-compressor (QuantizationModifier, scheme="NVFP4", ignore=["lm_head"]), sequential layer-wise calibration on a single GPU with the model resident in CPU RAM. NVIDIA ModelOpt was tried first and failed on this hybrid GDN architecture (both latest and pinned versions), llm-compressor handled it cleanly and auto-excluded the quantization-sensitive linear_attn (GDN) layers — same treatment as NVIDIA's official NVFP4 build.
—Calibration: 512 mixed-language samples — 256 English news (cnndailymail) + 256 Chinese instruction pairs (alpaca-zh), maxlength 1024.
—Multimodal graft: llm-compressor only saves the text model (which vLLM refuses to load for this architecture — only the wrapper arch is registered). We grafted the 333 BF16 vision-tower tensors from the upstream checkpoint back into the safetensors, rebuilt the wrapper-format config.json (visual modules listed explicitly in the quantization ignore — regex patterns break vLLM weight loading with a KeyError), and restored the processor config set.

Lineage & licensing

Qwen/Qwen3.6-27B (Apache-2.0) → rico03/Qwen3.6-27B-Claude-Opus-Reasoning-Distilled (Apache-2.0, 14k Claude Opus 4.6 reasoning traces — see upstream repo for training-data details) → this NVFP4 quantization (Apache-2.0). Vision tower and tokenizer inherit from the official base checkpoint.