pavanperi/Voxtral-Mini-4B-Realtime-2602_W4A16_G128

Voxtral-Mini-4B-Realtime-2602 — GPTQ W4A16 Compressed

Compressed checkpoint of `mistralai/Voxtral-Mini-4B-Realtime-2602` produced with llm-compressor using GPTQ W4A16 quantization on the decoder's linear layers.

1. Environment Setup (uv recommended)

Create the project

mkdir voxtral-serve && cd voxtral-serve

copy pyproject.toml in the directory

[project]
name = "voxtral-serve"
version = "0.1.0"
description = "Compress Voxtral-Mini-4B-Realtime-2602 using llmcompressor"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
    "vllm",
    "mistral-common>=1.5.4",
    "soundfile>=0.12",
    "librosa>=0.10",
    "soxr>=0.3",
]

Install everything in an isolated venv:

uv venv .venv --python 3.12
source .venv/bin/activate
uv sync

serve the compressed model

uv run vllm serve --config vllm_config.yaml

2. Compression technique

GPTQ W4A16

The script applies GPTQ inthe W4A16 configuration:

GPTQ isa one-shot, data-driven post-training quantization algorithm. For each Linear layer it:

1. 1.Collects the Hessian of the layer's reconstruction error using a small calibration dataset (256 audio samples here).
2. 2.Quantizes weight columns sequentially, ans it compensates the remaining unquantized columns using the Hessian inverse.
3. 3.Stores the resulting tensors.

• —W4 — weights are quantized to 4-bit integers (INT4), stored in a blocked/grouped layout.
• —A16 — activations remain in bfloat16 at inference time; only the matrix-multiply dequantizes weights on the fly.
• —Group size 128 — the 4-bit scale and zero-point factors are computed per group of 128 consecutive weight columns, which id a good balance between accuracy and model size.

What is quantized and what is not

Only the decoder `Linear` layers are compressed. We preserved all the audio components which are very sensitive to precision loss and quantizing those would be a minimal size gain.

3. Capabilities of the Compressed Model