useful-quants/Qwen3-VL-4B-Instruct-W4A16-BF16Vision

Qwen3-VL-4B-Instruct W4A16 BF16-Vision

Ampere-friendly serving build of `Qwen/Qwen3-VL-4B-Instruct`. Text-side linears are compressed-tensors W4A16. The vision tower and multimodal connector stay BF16.

Use

vllm serve useful-quants/Qwen3-VL-4B-Instruct-W4A16-BF16Vision \
    --served-model-name Qwen3-VL-4B-Instruct-W4A16-BF16Vision \
    --trust-remote-code \
    --dtype bfloat16 \
    --quantization compressed-tensors \
    --max-model-len 4096 \
    --limit-mm-per-prompt '{"image":1}' \
    --allowed-local-media-path / \
    --enforce-eager

Note: `--enforce-eager` is recommended in some vLLM versions to avoid buffer issues with deepstack tokens.

Smoke test

python verify.py --url http://localhost:8000/v1/chat/completions

Notes

• —Built for RTX 30xx/40xx Ampere cards.
• —Vision stays BF16 on disk and at runtime.
• —The repo includes the calibration corpus used for quantization.