Vishva007/Qwen3.5-9B-W4A16-AutoRound-GPTQ
Vishva007/Qwen3.5-9B-W4A16-AutoRound-GPTQ
This is a W4A16 (4-bit weight, 16-bit activation) quantized version of Qwen/Qwen3.5-9B, produced using AutoRound — Intel's sign gradient descent based quantization method designed for production-grade accuracy retention. MTP Enabled model quantization
Quantization Details
Key Notes
- High accuracy configuration — 1200 iterations with 512 calibration samples targets production-grade quality with minimal degradation from the base model.
- W4A16 — Weights are quantized to 4-bit integers; activations remain in FP16 for inference stability.
- ~50% memory reduction compared to the FP16 base model, enabling deployment on consumer and mid-range GPUs.
- Vision Tower (`quant_nontext_module`):
False(Kept in BF16 to preserve visual reasoning and OCR precision) - Special Modules (`layer_config`): Multi-Token Prediction (
mtp,mtp.fc) kept in native bfloat16
MTP / Speculative Decoding
This model supports Multi-Token Prediction (MTP) for improved inference throughput using speculative decoding.
When serving with compatible backends (e.g., vLLM), enable MTP using:
--speculative_config '{"method":"mtp","num_speculative_tokens":1}'Notes
num_speculative_tokens=1is a stable default for balancing speed and accuracy.- You can experiment with higher values for better throughput, depending on your hardware and latency requirements.
Usage
This model is compatible with transformers and backends that support AutoRound GPTQ-format weights (e.g., vLLM, SGLang, AutoGPTQ). For full model details, architecture, and capabilities, refer to the base model page.
🚀 Deploy on RunPod
One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.
🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.