CompressedMichael/Qwen3-VL-30B-A3B-Instruct-SmoothQuant-W8A8

Qwen3-VL-30B-A3B-Instruct — SmoothQuant-W8A8

Quantized derivative of Qwen/Qwen3-VL-30B-A3B-Instruct, created using a local llm-compressor checkout. SmoothQuant with smoothing strength 0.8, followed by INT8 rounding. All expert gate/up projections and the unquantized router are balanced against each post-attention norm to preserve routing under smoothing.

Language attention and all MoE expert linear weights are quantized. Vision weights, embeddings, router gates, normalization layers, and lm_head use BF16. The MoE stores all 30B-class model weights even though only a subset of experts is active for each token.

Calibration

128 shuffled Flickr30k test-split image/first-caption pairs, seed 42, with every expert receiving calibration inputs. Each example uses up to 256 image tokens and 1024 total tokens. Dataset revision: 765d117f3eec816f2bfdc2d73ebb50a6f77b86a4. Each quantization method starts independently from the original BF16 weights. The recipe and package versions are in recipe.yaml and quantization_run.json.

Load with Transformers

The source model's fused 3D expert parameters were converted to separate 2D linear modules. Use `AutoModelForImageTextToText` with `trust_remote_code=True` to load the bundled modeling_qwen3_vl_moe_quantized.py implementation. Loading directly with the stock Qwen3-VL MoE class does not reproduce this layout.

The validated environment used Torch 2.7.0, Transformers 4.57.1, compressed-tensors 0.13.0, and Accelerate. Transformers can decompress weights to BF16 while loading; allow memory for the full BF16 model plus working space. Saved checkpoint size is not a measurement of runtime GPU memory.

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "CompressedMichael/Qwen3-VL-30B-A3B-Instruct-SmoothQuant-W8A8"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="sdpa",
).eval()

This is a compressed-tensors export. Compatibility with AutoGPTQ serialization, vLLM serving, and accelerated low-bit kernels has not been validated.

Validation

On September 13, 2026, the local export passed a fresh Transformers reload with no missing, unexpected, or mismatched weights and no loader errors. Greedy generation of 48 tokens for the next held-out Flickr30k image produced a nonempty image description with no NaN generation scores. See validation.json. artifact_audit.json records quantization coverage, storage dtypes, shard index, and file completeness checks for all 18,624 quantized linears.

These checks establish reload and generation functionality. Benchmark accuracy, accuracy relative to the original model, and low-bit serving speed were not measured in this validation. The calibration and validation images are not redistributed in this model repository.

License and attribution

The original model is from the Qwen team and is released under Apache-2.0. This repository provides a quantized derivative under the same license. See the original model card, LICENSE, and NOTICE.