et0dev/Qwen3.5-122B-A10B-NVFP4-Full-GB10

Qwen3.5-122B-A10B-NVFP4-Full-GB10

All-NVFP4 (W4A4) quantization of Qwen/Qwen3.5-122B-A10B — the maximum-decode-speed variant for NVIDIA DGX Spark (GB10, SM121). Every Linear layer is 4-bit, including the lm_head. Vision encoder preserved (BF16) and verified working.

This is the most aggressively quantized checkpoint of this model: at GB10's memory bandwidth it decodes ~33–35 tok/s single-stream, ~45% faster than the mixed-precision FP8Dense sibling (~23–24 tok/s).

Quality note

The all-FP4 profile carries a subtle quality degradation relative to the FP8Dense sibling: the dense path (attention + GDN) runs 4-bit activations, and the lm_head's 4-bit weights flatten the output distribution slightly. It does not show up on simple extraction benchmarks (both variants ace detail tests), but in extended real-world use comprehension and nuance are noticeably better on FP8Dense. Pick by priority:

• —Maximum tok/s → this checkpoint
• —Best quality at ~2/3 the speed → FP8Dense

Layout

Excluded from quantization (BF16): router gates (mlp.gate, shared_expert_gate), embeddings, vision encoder + merger, norms. No MTP weights in this checkpoint (the config's MTP declaration has no corresponding tensors — speculative decoding is not available).

Quantization details

• —Method: llm-compressor oneshot(), NVFP4 scheme (W4A4, group 16, FP8-E4M3 scales)
• —Calibration: 512 samples, HuggingFaceH4/ultrachat_200k, seq_len 2048
• —Quantized from the full multimodal model (AutoModelForImageTextToText) so the vision tower survives — earlier text-only exports of this model lost it

Benchmarks (DGX Spark GB10, vLLM 0.19.2 from-source SM121 build)

Sequential decode, gen=500:

Concurrent @ 32K context, gen=500:

KV cache: 5.94× the full 262K context at --gpu-memory-utilization 0.90 (bf16 KV; page size 2,096 tokens, aligned to the GDN/Mamba state size).

Serving (vLLM)

vLLM's stock ParallelLMHead cannot load NVFP4-packed lmhead weights (it inherits `VocabParallelEmbedding`'s loader). The one-file patch in [`vllm-patches/patchnvfp4lmhead.py](./tree/main/vllm-patches) swaps the lm_head to ReplicatedLinear`, which routes through the standard quantized-linear loading path:

docker run -d --name vllm --gpus all -p 8000:8000 --ipc host \
  -v /opt/vllm-cache:/root/.cache/huggingface \
  -e CUBLASLT_WORKSPACE_SIZE=33554432 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False \
  vllm/vllm-openai:latest \
  --model demon-zombie/Qwen3.5-122B-A10B-NVFP4-Full-GB10 \
  --served-model-name Qwen3.5-122B-A10B \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

# apply the lm_head patch, then restart
docker cp patch_nvfp4_lm_head.py vllm:/tmp/
docker exec vllm python3 /tmp/patch_nvfp4_lm_head.py
docker restart vllm

Verified configuration: DGX Spark GB10, from-source vLLM 0.19.2 (SM121, TORCHCUDAARCHLIST 12.1) with the patch applied — all benchmark numbers above are from that build. Newer stock images have not been re-verified with this checkpoint's NVFP4 lmhead; if loading fails around lm_head/ParallelLMHead, apply the included patch.

Architecture

Qwen3.5-122B-A10B is a hybrid-attention MoE model: 48 layers (36 GDN/Gated-DeltaNet linear attention + 12 full attention, interval 4), 256 routed experts + 1 shared expert per layer (8 active), 3,072 hidden, 248,320 vocab, 262K context, plus a ViT vision encoder.

Credits

• —Base model: Qwen/Qwen3.5-122B-A10B
• —Quantized with llm-compressor
• —Sibling checkpoint: FP8Dense (mixed precision, recommended for quality)