Shankara-A-S/voxtral-mini-4b-realtime-gguf

Voxtral-Mini-4B-Realtime-2602 — GGUF (llama.cpp)

Quantized GGUF files for mistralai/Voxtral-Mini-4B-Realtime-2602.

Converted using the didlawowo/llama.cpp feat/voxtral-realtime-clean fork which implements the Voxtral Realtime dual-stream inference protocol.

Files

Requirements

Clone and build the voxtral-realtime fork of llama.cpp:

git clone --depth=1 --branch feat/voxtral-realtime-clean \
  https://github.com/didlawowo/llama.cpp.git llama-voxtral
cd llama-voxtral
cmake -B build-cuda -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build-cuda -j --target llama-server

Usage

# Download model files
hf download Shankara-A-S/voxtral-mini-4b-realtime-gguf

# Start server (IMPORTANT: use -c 2048 --parallel 1)
./build-cuda/bin/llama-server \
  -m voxtral-rt-dec-Q4_K_M.gguf \
  --mmproj voxtral-rt-mmproj.gguf \
  --port 8080 \
  -ngl 99 \
  -c 2048 \
  --parallel 1

Then call the OpenAI-compatible transcription endpoint:

curl -X POST http://localhost:8080/v1/audio/transcriptions \
  -F file=@audio.wav \
  -F model=whisper-1

Response:

{"task":"transcribe","language":"en","duration":0.0,"text":"..."}

Model Architecture

• —Decoder: 26-layer Mistral3 GQA (dim=3072, heads=32/8, ff=9216) with per-layer adaptive RMSNorm (FiLM conditioning via 32-dim delay token embeddings)
• —Encoder: 32-layer causal transformer (dim=1280, heads=32, sliding_window=750, RoPE, SwiGLU)
• —Projector: 2-layer MLP with 4× stack downsampling
• —Vocab: 131072 tokens (Tekken tokenizer)

Quality (local RTX 5080 test)

LibriSpeech test-clean: Q4KM transcription is identical to BF16 output.

Notes

• —The server must use -c 2048 --parallel 1 for audio transcription. With default settings (n_parallel=4, c=512), each slot gets only 128 tokens—not enough for the ~180 audio tokens per 5s clip.
• —The audio encoder GGUF (voxtral-rt-mmproj.gguf) is BF16 to preserve encoder quality.
• —Adaptive RMSNorm tensors (blk.X.ffn_ada_norm_up.weight) are loaded as optional; the fork uses the precomputed default-delay scale.