mastersubhajit/sarvam-30b-AWQ-4bit

sarvam-30b-AWQ-4bit

This model is a quantized version of the Sarvam-30B multilingual Mixture-of-Experts (MoE) large language model. It was compressed using Activation-Aware Weight Quantization (AWQ) into a 4-bit representation at the National Electronics and Computer Technology Center (NECTEC), Thailand, in collaboration with the Asian Institute of Technology (AIT).

The primary objective of this model is to substantially reduce the GPU memory and storage requirements needed to run the 32-billion parameter MoE architecture, making it highly suitable for consumer GPUs and resource-constrained inference environments while preserving multilingual reasoning and language capabilities.

Another more advanced quantized model is available, the Sarvam-30B-AWQ-GPTQ-Hybrid is caliberated on multiple datasets which seems to perform slightly better compared the regular quantized models.

Model Details

• —Model Creator: Subhajit Ghosh (mastersubhajit)
• —Institution: NECTEC (National Science and Technology Development Agency) / Faculty of Advanced Science & Technology (FAST), AIT
• —Base Model: Sarvam-30B (BF16)
• —Architecture: sarvam_moe (Mixture-of-Experts)
• —Quantization Scheme: 4-bit weight, 16-bit runtime activation precision (W4A16)
• —Quantization Method: AutoAWQ (Weight-only post-training quantization)
• —Group Size: 128
• —Zero-Point Quantization: Enabled
• —Weight-Packing Format: GEMM
• —Calibration Datasets: IndiVibe (512 samples; sequences > 256 characters; max sequence length 4,096 tokens).

Quantization Methodology

The standard sarvam_moe sparse MoE structure requires customized handling for quantization to preserve the model's routing stability and output quality.

Activation-Aware Scaling

The model was quantized using standard Activation-Aware Weight Quantization (AWQ), which leverages activation information from the calibration dataset to identify channels that contribute strongly to model outputs. Scaling was applied to reduce the error associated with important channels instead of quantizing all weights identically.

Precision Retention Strategy (Mixed Precision)

To maintain structural and reasoning integrity, not all layers were converted to 4-bit. Sensitive components were explicitly retained in higher precision (BF16):

• —Layer 0: Retained entirely in BF16, as early layers are particularly sensitive to quantization error.
• —MoE Routers (`mlp.gate`): Retained in BF16 to ensure that the expert selection path is completely preserved without numerical interference.
• —LM Head: Retained in BF16 to preserve vocabulary logit precision.
• —Quantized Modules: Supported attention projections, routed expert projections, and shared expert projections in Layers 1–18 were successfully quantized to 4-bit.

Evaluation Results

The AWQ model was evaluated alongside the original BF16 model using a strict conditional log-probability scoring method (to eliminate generation/sampling variance) on the LANTA HPC system.

(Note: The chain-of-thought GSM8K result for the quantized model exceeded the BF16 baseline under this specific evaluation protocol, likely due to prompt sensitivity or evaluation variance rather than an intrinsic improvement in mathematical reasoning.)

Key Takeaways

• —The 4-bit AWQ model successfully retains ~98.6% of the BF16 MMLU score and ~95.7% of the GPQA Diamond score.
• —For multilingual understanding, it retains over 98% of the BF16 performance on the Hindi ARC-Indic dataset.
• —Significant reduction in storage size and runtime VRAM usage was achieved, making single-node/consumer-GPU inference highly practical.

Inference and Deployment

This model is configured for GEMM weight-packing, meaning it is specifically optimized for batched evaluation and OpenAI-compatible inference serving.

During testing, the model successfully demonstrated its ability to:

• —Be served interactively via vLLM's OpenAI-compatible routing environment.
• —Perform reliable instruction-following, spelling error correction, and elementary arithmetic logic.
• —Process streaming token generation securely.

Intended Use

This 4-bit model serves as a practical solution for developers and institutions wishing to run the Sarvam-30B architecture in environments constrained by GPU memory. It sacrifices a minimal amount of aggregate accuracy for massive gains in deployment flexibility and reduced inference costs.

Acknowledgements

Developed by Subhajit Ghosh at the National Electronics and Computer Technology Center (NECTEC), Thailand.