mistral-experimental/AudioCPP-Voxtral-Mini-4B-Realtime-2602-GGUF

Voxtral Mini 4B Realtime GGUF for audio.cpp

This repository contains quantized standalone GGUF checkpoints for running Voxtral Mini 4B Realtime ASR with audio.cpp. The GGUF files embed the audio.cpp model spec and required sidecars, so the model can be used directly from the checkpoint path without a separate local model-spec directory.

What audio.cpp does

audio.cpp is a young C++/GGML audio inference framework focusing on CUDA performance. It runs speech and audio models locally with CLI and server interfaces, including ASR, TTS, voice conversion, source separation, diarization, VAD, and audio generation models. The project currently tracks 40+ model families and is growing quickly. For Voxtral Realtime, audio.cpp provides offline and streaming speech recognition from a single GGUF checkpoint.

Files

Download

Each checkpoint is a single self-contained file, so download only the quantization you intend to run rather than the whole repository.

Q8_0 (default):

hf download mistral-experimental/AudioCPP-Voxtral-Mini-4B-Realtime-2602-GGUF \
  voxtral-mini-4b-realtime-2602-q8_0.gguf \
  --local-dir ./voxtral-realtime-gguf

Q4_K (smaller and faster):

hf download mistral-experimental/AudioCPP-Voxtral-Mini-4B-Realtime-2602-GGUF \
  voxtral-mini-4b-realtime-2602-q4_k.gguf \
  --local-dir ./voxtral-realtime-gguf

Either lands at ./voxtral-realtime-gguf/<filename>.gguf, which is the path to pass as --model.

The hf command ships with huggingface_hub (pip install -U huggingface_hub). On releases before the CLI was renamed, the same call is huggingface-cli download.

Without Python, fetch the file over HTTP instead:

curl -L -o voxtral-mini-4b-realtime-2602-q8_0.gguf \
  https://huggingface.co/mistral-experimental/AudioCPP-Voxtral-Mini-4B-Realtime-2602-GGUF/resolve/main/voxtral-mini-4b-realtime-2602-q8_0.gguf

Point --model at the .gguf file itself, not at the folder holding it. audio.cpp accepts a directory only when it can narrow it to a single GGUF, so a folder containing both quantizations is rejected rather than guessed at:

model directory contains 2 GGUF files: ./voxtral-realtime-gguf; found: ...
pass one of them directly with --model, or keep a single GGUF in the directory

Performance

Measurements below are from audio.cpp CUDA validation runs. Results vary by GPU, driver, backend, audio length, and decode settings.

Q8_0 vs BF16 reference

BF16 was used only as a local reference baseline for validation. This repository publishes quantized GGUF checkpoints only.

Q4_K quick check

In the quick validation set, Q4K transcripts matched Q80 except for one capitalization-only difference.

Use with audio.cpp

Build audio.cpp with the helper script for your platform, then use the generated audiocpp_cli binary. The project provides build paths for Linux, Windows, and macOS; see the audio.cpp README for the current build matrix and detailed requirements.

git clone https://github.com/0xShug0/audio.cpp
cd audio.cpp

# Linux: CUDA, Vulkan, or CPU
scripts/build_linux.sh --backend cuda --target audiocpp_cli

# Windows: CUDA or CPU presets
powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\scripts\build_windows.ps1 -Preset windows-cuda-release -Target audiocpp_cli

# macOS: Metal
scripts/build_metal.sh --target audiocpp_cli

The examples below assume audiocpp_cli is on your PATH. You can also replace it with the built binary path for your platform, such as build/linux-cuda-release/bin/audiocpp_cli. They pass --backend cuda; use --backend metal on macOS, or --backend cpu when no GPU backend is built.

Run offline ASR with Q8_0:

MODEL=/path/to/Voxtral-Mini-4B-Realtime-2602-GGUF/voxtral-mini-4b-realtime-2602-q8_0.gguf

audiocpp_cli \
  --task asr \
  --family voxtral_realtime \
  --model "$MODEL" \
  --backend cuda \
  --threads 8 \
  --audio input.wav \
  --text-out transcript.txt

Run the Q4_K checkpoint by changing the model path:

MODEL=/path/to/Voxtral-Mini-4B-Realtime-2602-GGUF/voxtral-mini-4b-realtime-2602-q4_k.gguf

Streaming ASR

While a stream runs, the CLI reports each update as the text decoded since the previous one, so the updates concatenate into the transcript. On a terminal they are appended and the transcript scrolls like ordinary output; when stdout is redirected, each update is written as its own flushed partial_text= line so pipes and logs stay parseable. --text-out and the final text_output= line carry the complete transcript either way.

Streaming from an audio file:

audiocpp_cli \
  --task asr \
  --family voxtral_realtime \
  --model "$MODEL" \
  --backend cuda \
  --threads 8 \
  --mode streaming \
  --audio input.wav \
  --text-out transcript.txt

Streaming raw 16 kHz mono PCM from ffmpeg:

ffmpeg -i input.mp3 -ar 16000 -ac 1 -f s16le - \
  | audiocpp_cli \
      --task asr \
      --family voxtral_realtime \
      --model "$MODEL" \
      --backend cuda \
      --threads 8 \
      --mode streaming \
      --audio -

Going faster with stream_batch_tokens

A streaming step always advances 80 ms of audio, so each step has to cost under 80 ms to keep pace with a live source. A step is one frontend pass, one audio-encoder forward, and one text-decoder step. Setting voxtral_realtime.stream_batch_tokens=<n> makes a single encoder forward cover n audio tokens instead of one. The decoder still runs once per token, so batching amortizes the encoder's fixed per-dispatch cost and leaves decode work untouched:

audiocpp_cli \
  --task asr \
  --family voxtral_realtime \
  --model "$MODEL" \
  --backend cuda \
  --threads 8 \
  --mode streaming \
  --audio input.wav \
  --session-option voxtral_realtime.stream_batch_tokens=4

Measured on an Apple M3 MacBook Air with Metal and Q8_0, over a 14 s clip (169 steps, mean of three runs). These are laptop numbers and are not comparable to the CUDA figures earlier in this card; what carries across hardware is the split between the two middle columns.

The decoder column is flat, which is the whole shape of this knob: batching can only remove encoder time, so the decoder sets a floor and the gain flattens once the encoder is no longer the bottleneck. 2 is not worth taking — it barely moves the encoder — while 4 captures most of the available win.

What you pay for it is latency. A chunk is not transcribed until all n of its audio tokens have arrived, so each update is delayed by up to n * 80 ms: 0.32 s at 4, 0.64 s at 8. Transcribing a file, that delay costs nothing perceptible; on a live microphone it is the tradeoff to weigh.

Note also that the decoder runs one step per 80 ms whether the audio holds speech or silence, so a session that does fall behind stays behind — the lag does not recover during pauses. Measure your own hardware before relying on a live source.

Prompting and decoding

audio.cpp exposes Voxtral Realtime as an ASR model. Pass audio to the CLI and audio.cpp builds the model's transcription prompt internally; no free-form text prompt is required for normal transcription.

Decode options can be controlled from the request:

audiocpp_cli \
  --task asr \
  --family voxtral_realtime \
  --model "$MODEL" \
  --backend cuda \
  --threads 8 \
  --audio input.wav \
  --text-out transcript.txt \
  --request-option max_new_tokens=256 \
  --do-sample false \
  --temperature 1.0 \
  --top-p 1.0 \
  --top-k 50 \
  --seed 1234

Notes

• —Task: asr
• —Family: voxtral_realtime
• —Supported modes: offline and streaming
• —Timestamp output is not currently exposed by audio.cpp for this model
• —The GGUF package is intended to be standalone: model weights, sidecars, and audio.cpp model spec are embedded in the checkpoint