juanginer/Voxtral-Mini-3B-2507-AWQ4-GEN

Note on parameter count. Hugging Face reports ~5B parameters from the safetensors metadata, inflated due to how AWQ4 stores each group of 8 int4 weights packed into a single int32 tensor. Actual disk and VRAM footprint of the model is roughly (4.3 GB).

AWQ4-GEN

General-purpose AWQ4 W4A16 quant of Voxtral-Mini-3B-2507, calibrated across 8 European languages.

When to pick this variant

Use this if you serve mixed-language audio and want the safest all-around AWQ4. Matches the bf16 baseline on Polish, near-parity on Spanish and English, and 2x faster.

Benchmarks vs bf16 baseline

Data: FLEURS + Common Voice 15 + Multilingual LibriSpeech + ylacombe/google-argentinian-spanish (es-AR). Hardware: 2x RTX 4090, vLLM 0.25. Bold = best 4-bit variant.

Usage (vLLM)

# 1. In your vLLM venv, add the deps Voxtral needs
uv pip install --python /path/to/vllm/venv/python "mistral-common[soundfile]"

# 2. Serve
vllm serve JuanGiner/AWQ4-GEN \
    --tokenizer-mode mistral \
    --config-format mistral \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.9 \
    --port 8000

Then send audio to POST /v1/audio/transcriptions in OpenAI-compatible format.

Calibration

480 audio samples symmetric across 8 languages (60 per language via google/fleurs + ylacombe/google-argentinian-spanish). Fixes the Polish regression seen in the text-calibrated AWQ4.

Quantization was done with llmcompressor 0.10 (AWQ modifier, W4A16 scheme, group_size=128, duo_scaling=True), targeting Linear modules inside LlamaDecoderLayer. The compressed-tensors format matches what vLLM 0.25.x expects for AWQ models (weights re-fused for the vLLM Voxtral loader).

Base model & license

Base: `mistralai/Voxtral-Mini-3B-2507`, Apache-2.0. This quantized derivative is released under the same license.