Vishva007/Qwen3.8-4B-Distill-W4A16-AutoRound-LLM-Compressor

Qwen3.8-4B-Distill (W4A16 Quantized via AutoRound)

This repository contains a W4A16 (4-bit weights, 16-bit activations) quantized version of empero-ai/Qwen3.8-4B-Distill, quantized using Intel's AutoRound algorithm.

⚡ Quantization Details

Calibrated and quantized with fine-grained group sizes and high iteration depth to preserve reasoning traces (<think> blocks) and multimodal capabilities:

• —Algorithm: Intel AutoRound
• —Precision / Scheme: W4A16 (4-bit weights, 16-bit activations)
• —Group Size: 32 (fine-grained reconstruction fidelity)
• —Symmetric (`sym`): True
• —Calibration Samples (`nsamples`): 512
• —Sequence Length (`seqlen`): 4096
• —Tuning Iterations (`iters`): 1000 (Production-grade accuracy)
• —Vision Tower (`quant_nontext_module`): False (Kept in BF16 to preserve visual reasoning and OCR precision)
• —Special Modules (`layer_config`): Multi-Token Prediction (mtp, mtp.fc) kept in native bfloat16

📦 Available Formats

Depending on your inference engine, choose the appropriate repository:

• —AutoRound Format: `Vishva007/Qwen3.8-4B-Distill-W4A16-AutoRound`
• —AutoGPTQ Format: `Vishva007/Qwen3.8-4B-Distill-W4A16-AutoRound-GPTQ`
• —LLM-Compressor / Compressed-Tensors Format: `Vishva007/Qwen3.8-4B-Distill-W4A16-AutoRound-LLM-Compressor`

🚀 Usage & Quickstart

1. High-Throughput Serving via vLLM

# Using the LLM-Compressor / Compressed-Tensors build
vllm serve Vishva007/Qwen3.8-4B-Distill-W4A16-AutoRound-LLM-Compressor \
    --dtype bfloat16 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.90

📊 VRAM & Performance Benefits

• —Original Model (BF16): ~8–10 GB VRAM required for full context inference
• —Quantized Model (W4A16 Group 32): ~2.5–3.5 GB VRAM (runs comfortably on 4GB/6GB consumer GPUs, laptops, and edge devices)
• —Throughput: Lowers memory bandwidth pressure, accelerating token generation speeds during extended chain-of-thought (<think>) reasoning.

🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.

PyTorch 2.14

PyTorch 2.13

PyTorch 2.12

📚 Acknowledgments

• —Original Distilled Model: Developed by Empero AI
• —Base Architecture: Qwen Team (Alibaba)
• —Quantization Framework: Intel AutoRound