Modelpublic

shunyang90/Qwen3-Omni-30B-A3B-Instruct-NVFP4

sourceHugging Faceapache-2.0updated 6mo agoView on Hugging Face

1likes253downloads

Model Card

Qwen3-Omni-30B-A3B-Instruct NVFP4 (W4A8)

Pre-quantized NVFP4 version of Qwen/Qwen3-Omni-30B-A3B-Instruct for deployment on NVIDIA Blackwell GPUs (RTX 5090, B100, B200).

Key Features

—4-bit weight, 8-bit activation (W4A8) quantization using NVIDIA ModelOpt NVFP4
—Pre-quantized checkpoint — weights are packed as FP4 (uint8), not online quantization
—~3.6x compression on MoE expert weights (54 GB → 15 GB)
—Same capabilities as the original: text, image, audio, video input → text + speech output

Quantization Details

Component	Precision	Quantized?
Thinker MoE experts (gateupproj, down_proj)	NVFP4 packed uint8	Yes — pre-quantized
Thinker attention (q/k/v/o_proj)	NVFP4 (calibrated)	Yes
Thinker lm_head	BF16	No
Thinker MoE router gates	BF16	No
Audio Encoder	BF16	No
Vision Encoder	BF16	No
Talker (MoE)	BF16	No
Code2Wav	BF16	No

Weight Format

MoE expert weights are stored as:

—gate_up_proj: packed uint8 (2× FP4 values per byte)
—gate_up_proj_scale: float8_e4m3fn (per-block-of-16 FP8 scale)
—gate_up_proj_scale_2: bfloat16 (global per-tensor scale)
—Same pattern for down_proj

Quantization Config

json

{
  "quant_algo": "NVFP4",
  "quant_method": "modelopt",
  "group_size": 16,
  "weights": {"num_bits": 4, "type": "float"},
  "input_activations": {"num_bits": 8, "type": "float"},
  "kv_cache": {"num_bits": 8, "type": "float"}
}

Memory Comparison

Config	Checkpoint Size	Notes
BF16 (original)	~60 GB	Full precision
FP8 (ModelOpt)	~40 GB	Online quantization
NVFP4 (this model)	~27 GB	Pre-quantized FP4 experts

Hardware Requirements

Purpose	Hardware	Notes
Inference	RTX 5090, B100, B200 (Blackwell SM 100+)	Native FP4 hardware support required
Inference (not supported)	RTX 4090, A100, H100	No FP4 hardware — use FP8 instead

Quantization Methodology

1.Base model: Qwen/Qwen3-Omni-30B-A3B-Instruct (BF16)
2.Tool: NVIDIA ModelOpt v0.42.0
3.Method: NVFP4 (W4A8) with static calibration (256 diverse prompts)
4.Calibration: 256 English + Chinese prompts covering factual, creative, code, math, and reasoning tasks
5.Attention layers: Quantized via mtq.quantize with calibration
6.Expert layers: Manually packed via NVFP4QTensor.quantize() (fused expert tensors not supported by mtq.quantize)
7.Excluded: lm_head, MoE router gates, all non-thinker stages

Original Model

This is a quantized version of Qwen/Qwen3-Omni-30B-A3B-Instruct. Please refer to the original model card for full capabilities, benchmarks, and usage documentation.

License

Apache License 2.0 (same as the original model)