cstr/voxtral-mini-4b-realtime-GGUF

Voxtral-Mini-4B-Realtime — GGUF

GGUF quantizations of mistralai/Voxtral-Mini-4B-Realtime-2602, a 4.4B-parameter realtime streaming speech-to-text model with a causal audio encoder and configurable transcription delay.

Converted and tested with CrispASR, a multi-model ASR framework built on ggml.

Files

⚠ This table used to name the F16 voxtral-mini-4b-realtime.gguf, and no such file was ever uploaded. The published name is …-f16.gguf, matching the quant suffix CrispASR's -m auto:f16 resolver expects.

Performance (CPU, 4 threads, AVX2, jfk.wav 11s)

Q4_K recommended — 3.5× smaller than F16, 2.7× faster, identical transcription quality.

Usage

# Build CrispASR
git clone https://github.com/CrispStrobe/CrispASR
cd CrispASR
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc) --target voxtral4b-main

# Download Q4_K (recommended)
huggingface-cli download cstr/voxtral-mini-4b-realtime-GGUF \
    voxtral-mini-4b-realtime-q4_k.gguf --local-dir .

# Transcribe
./build/bin/voxtral4b-main -m voxtral-mini-4b-realtime-q4_k.gguf -f audio.wav

With word-level timestamps

# Also download the CTC aligner
huggingface-cli download cstr/canary-ctc-aligner-GGUF \
    canary-ctc-aligner-q4_k.gguf --local-dir .

./build/bin/voxtral4b-main -m voxtral-mini-4b-realtime-q4_k.gguf \
    -f audio.wav -am canary-ctc-aligner-q4_k.gguf -timestamps

CLI options

-m  FNAME   GGUF model file (required)
-f  FNAME   Input audio, 16 kHz mono WAV (required)
-t  N       Threads (default: 4)
-l  LANG    Language hint (default: en)
-n  N       Max new tokens (default: 512)
-am FNAME   CTC aligner GGUF for word timestamps
-timestamps Enable word-level timestamps (requires -am)
-np         Suppress stderr info

Architecture

• —Audio encoder: 32-layer causal transformer (RoPE, SwiGLU, RMSNorm, sliding window 750)
• —LLM decoder: 26-layer Mistral (GQA 32/8, SwiGLU, adaptive RMSNorm, sliding window 8192)
• —Projector: 4-frame stack → Linear(5120→3072) → GELU → Linear(3072→3072)
• —Tokenizer: Mistral Tekken (150K vocab, 1000 special tokens)
• —Audio injection: adapter output ADDED to token embeddings (streaming format)

Key features

• —Natively streaming architecture with causal encoder
• —13 languages: en, fr, es, de, ru, zh, ja, it, pt, nl, ar, hi, ko
• —Configurable delay: 480ms default (6 tokens × 80ms)
• —Apache 2.0 license

Conversion

python models/convert-voxtral4b-to-gguf.py \
    --input /path/to/Voxtral-Mini-4B-Realtime-2602 \
    --output voxtral-mini-4b-realtime-f16.gguf

# Then quantize
./build/bin/crispasr-quantize voxtral-mini-4b-realtime-f16.gguf \
    voxtral-mini-4b-realtime-q4_k.gguf q4_k

Credits

• —Model: Mistral AI — Apache 2.0
• —GGUF conversion: CrispASR
• —Port cross-referenced against voxtral.c, voxmlx, voxtral-mini-realtime-rs

Provenance and EU AI Act Art. 53 note

• —Upstream model: mistralai/Voxtral-Mini-4B-Realtime-2602 — published by mistralai.
• —Upstream licence: apache-2.0. This repository redistributes under the same terms; it grants no rights the upstream licence does not.
• —What was done here: format conversion and/or quantisation only (GGUF). No training, no fine-tuning, no merging, no distillation, no change to architecture, vocabulary or capability. Only the numeric representation of the upstream weights differs.
• —Training data: documented — where it is documented at all — by the upstream provider; see the upstream model card. No training data was used, added or selected by this repository. No training-content summary was found on the upstream model card at the time of writing; that documentation gap is upstream's and is not filled here.
• —Provider status: under Regulation (EU) 2024/1689 the upstream authors remain the provider of this model. Converting the serialisation format does not make this repository the provider of a new general-purpose AI model, and no such claim is made. Questions about training content, copyright policy or model capability belong upstream.