s-batman/Qwen3.6-27B-NVFP4-MTP-GGUF

s-batman/Qwen3.6-27B-NVFP4-MTP-GGUF

NVFP4 quantization of Qwen3.6-27B with Multi-Token Prediction (MTP) heads, converted to GGUF format using llama.cpp.

This quantization is specifically optimised for NVIDIA Blackwell consumer/edge GPUs (sm120/sm121) such as the RTX 5090 and DGX Spark (GB10). NVFP4 uses NVIDIA's native 4-bit block floating point format with E4M3 scaling, providing significantly faster inference than standard Q4_K quants on Blackwell hardware due to hardware-native dequantization.

Model Creator

Qwen Team (Alibaba Cloud)

Original Model

Qwen/Qwen3.6-27B — MTP variant based on unsloth/Qwen3.6-27B-GGUF

Quantization Details

What is NVFP4?

NVFP4 is NVIDIA's native 4-bit floating point format for Blackwell GPUs. Unlike standard integer quantization (Q4K, Q5K, etc.), NVFP4 uses block floating point with E4M3 scale factors and is dequantized directly by the GPU's tensor cores. This means:

• —Faster inference: Hardware-native dequantization eliminates the integer-to-float conversion overhead
• —Lower memory bandwidth: 4.60 BPW vs 10.47 BPW (Q8KXL) — 2.3× less data per token
• —Good quality: NVFP4 uses per-sub-block scaling (32 elements per sub-block) which preserves more information than uniform 4-bit quantization

MTP (Multi-Token Prediction)

This model includes the MTP prediction head from Qwen3.6, enabling speculative decoding with draft-mtp in llama.cpp. The MTP head is kept in Q4_K to preserve draft quality while the body uses NVFP4 for maximum throughput.

Performance

Benchmarked on NVIDIA DGX Spark (GB10, sm_121, 128 GB LPDDR5X, 273 GB/s bandwidth):

The 2× speedup comes directly from the reduced memory bandwidth requirement — the GB10's 273 GB/s LPDDR5X is the bottleneck for dense models, and NVFP4 halves the data transfer per token.

Provided Files

Usage with llama.cpp

Requirements

• —llama.cpp build 8967 or later (NVFP4 support merged in PR #22673)
• —CUDA toolkit with Blackwell (sm120/sm121) support
• —Build with -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=121 (adjust for your GPU)

Server

llama-server \
  -m Qwen3.6-27B-NVFP4-MTP.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  -c 262144 \
  -ngl 99 \
  -np 1 \
  -fa on \
  -ctk q8_0 -ctv q8_0 \
  --kv-unified \
  --no-mmap \
  --mlock \
  --cont-batching \
  --spec-type draft-mtp,ngram-mod \
  --spec-draft-n-max 3 \
  --spec-ngram-mod-n-match 24 \
  --spec-ngram-mod-n-min 4 \
  --spec-ngram-mod-n-max 48 \
  --temp 0.6 \
  --top-p 1 \
  --top-k 20 \
  --min-p 0.01 \
  --repeat-penalty 1.1

CLI

llama-cli \
  -m Qwen3.6-27B-NVFP4-MTP.gguf \
  -p "Explain quantum computing in simple terms" \
  -ngl 99 \
  --temp 0.6 \
  --top-p 1 \
  --top-k 20 \
  --min-p 0.01

Download with llama.cpp

llama-cli --hf-repo s-batman/Qwen3.6-27B-NVFP4-MTP-GGUF --hf-file Qwen3.6-27B-NVFP4-MTP.gguf -p "Hello"

Important Notes

• —Blackwell only: NVFP4 is a hardware-specific format. It will not run efficiently on non-Blackwell GPUs. For AMD, Intel, or older NVIDIA GPUs, use standard quantizations (Q4KM, Q5KM, etc.) from unsloth/Qwen3.6-27B-GGUF.
• —`--no-mmap` recommended: On unified memory architectures (DGX Spark), mmap can cause severe slowdowns.
• —`-np 1` required for MTP: Multi-token prediction speculative decoding currently requires single-parallel mode.

Licensing

This model is licensed under Apache 2.0, same as the original Qwen3.6-27B model. See LICENSE for details.