younghan-meta/Voxtral-4B-TTS-2603-ExecuTorch-CUDA

Voxtral-4B-TTS-2603-ExecuTorch-CUDA

Pre-exported ExecuTorch artifacts for Voxtral-4B-TTS-2603 with CUDA backend (NVIDIA GPU), bf16 precision, and 4-bit weight-only quantization (tile_packed_to_4d packing for _weight_int4pack_mm) on the LM decoder + flow head linears. The codec decoder is exported in fp32 via the same CUDA backend using a conv-as-matmul reformulation so it lowers onto Triton's batched-matmul kernels.

Overview

The pipeline has two stages: export (Python, once) and inference (C++ runner, repeated). This repo ships the export outputs so you can skip straight to inference.

The model has three components:

1. 1.Mistral 4B LLM decoder — autoregressive text → hidden states
2. 2.Flow Matching Head (3-layer transformer) — hidden states → 37 audio codebook tokens per frame via a 7-step Euler ODE
3. 3.Codec Decoder (Conv1d / ConvTranspose1d stack + 8 transformer layers) — codebook tokens → 24 kHz waveform

Pick the right folder for your GPU

ExecuTorch's CUDA backend uses AOTInductor, which *bakes pre-compiled cubins for the export-time GPU's compute capability into `.ptd**. The cubins are not forward/backward compatible across architectures — running an sm_80 blob on a Blackwell card fails with CUDA driver error: invalid argument` on the first kernel launch.

This repo ships per-arch artifacts in subfolders:

Find your GPU's compute capability with:

nvidia-smi --query-gpu=name,compute_cap --format=csv

If your arch isn't listed, re-export on your GPU.

Performance

End-to-end on seed=42, after the per-process Triton autotune cache is warm (first call is ~30–50 s slower; the runner's warmup() amortizes it).

Numerical parity vs the XNNPACK FP32 baseline (measured on the A100 export):

• —Last-position prefill hidden cosine: 0.999994
• —First-frame semantic argmax + top-5: identical

Artifacts target Linux NVIDIA GPUs.

Prerequisites

• —Linux with NVIDIA GPU + CUDA 12.8 or 12.9 toolkit (CUDA 13 not supported — ExecuTorch's backends/cuda/runtime/shims/sort.cu was written against CUB 2.x).
• —Conda + Python 3.10–3.12.
• —ExecuTorch built from source with the CUDA backend (see Build).
• —Tokenizer + at least one voice embedding from the upstream Mistral repo (~33 MB total). They are not included in this repo.

Install + Build

git clone https://github.com/pytorch/executorch/ ~/executorch
cd ~/executorch

unset CPATH                              # avoids CUDA-13 host header pollution
./install_executorch.sh
pip install -e . --no-build-isolation    # editable so source edits take effect

# Build runner with CUDA enabled
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH
make voxtral_tts-cuda

Binary lands at cmake-out/examples/models/voxtral_tts/voxtral_tts_runner.

WSL2 only: the linker also needs libcuda.so from the driver dir:

export LIBRARY_PATH=/usr/lib/wsl/lib:/usr/local/cuda/lib64/stubs:$LIBRARY_PATH
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:/usr/lib/wsl/lib:$LD_LIBRARY_PATH

Download

pip install huggingface_hub

# 1. Pre-exported ExecuTorch artifacts for your GPU arch (~3.7 GB).
#    Replace `sm80` with `sm120` for Blackwell (RTX 5080/5090).
hf download younghan-meta/Voxtral-4B-TTS-2603-ExecuTorch-CUDA \
    --include 'sm80/*' \
    --local-dir voxtral_tts_cuda

# 2. Tokenizer + voice embeddings from the base model (~33 MB)
hf download mistralai/Voxtral-4B-TTS-2603 \
    tekken.json voice_embedding/* \
    --local-dir voxtral_tts_base

Run

unset CPATH
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH

# Pick your arch
ARCH=sm80                                # or sm120

cmake-out/examples/models/voxtral_tts/voxtral_tts_runner \
    --model voxtral_tts_cuda/$ARCH/model.pte \
    --data_path voxtral_tts_cuda/$ARCH/aoti_cuda_blob.ptd \
    --codec voxtral_tts_cuda/$ARCH/codec_decoder.pte \
    --codec_data_path voxtral_tts_cuda/$ARCH/codec_aoti_cuda_blob.ptd \
    --tokenizer voxtral_tts_base/tekken.json \
    --voice voxtral_tts_base/voice_embedding/neutral_female.pt \
    --text "Hello, how are you today?" \
    --output output.wav \
    --seed 42 \
    --max_new_tokens 200

Output is 24 kHz mono 16-bit PCM. Listen with ffplay output.wav or aplay output.wav.

Streaming mode

Add --streaming to emit codec output in chunks instead of one batch at the end. The first chunk is always 9 600 samples (0.4 s) for low time-to-first- audio; subsequent chunks are 48 000 samples (2 s) until END_AUDIO. Pair with --speaker to pipe raw f32le PCM to stdout for live playback:

cmake-out/examples/models/voxtral_tts/voxtral_tts_runner \
    --model voxtral_tts_cuda/$ARCH/model.pte \
    --data_path voxtral_tts_cuda/$ARCH/aoti_cuda_blob.ptd \
    --codec voxtral_tts_cuda/$ARCH/codec_decoder.pte \
    --codec_data_path voxtral_tts_cuda/$ARCH/codec_aoti_cuda_blob.ptd \
    --tokenizer voxtral_tts_base/tekken.json \
    --voice voxtral_tts_base/voice_embedding/neutral_female.pt \
    --text "The quick brown fox jumps over the lazy dog." \
    --seed 42 \
    --streaming \
    --speaker \
  | ffplay -f f32le -ar 24000 -ac 1 -nodisp -autoexit -

For aplay instead: ... | aplay -f FLOAT_LE -r 24000 -c 1.

On RTX 5080, time-to-first-audio is ~2.6 s on a warm cache (most of which is the LM prefill); subsequent chunks arrive every ~0.5 s, comfortably ahead of playback.

Available voices

neutral_female, neutral_male, casual_female, casual_male, cheerful_female, ar_male, de_female, de_male, es_female, es_male, fr_female, fr_male, hi_female, hi_male, it_female, it_male, nl_female, nl_male, pt_female, pt_male — under `voice_embedding/` in the base-model repo.

Runner options

Re-exporting for another GPU

If your GPU's compute capability isn't shipped above, run the export on the target GPU (the AOTI compile step writes cubins for the local arch):

unset CPATH
export LIBRARY_PATH=/usr/lib/wsl/lib:/usr/local/cuda/lib64/stubs:$LIBRARY_PATH  # WSL only
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:/usr/lib/wsl/lib:$LD_LIBRARY_PATH

python examples/models/voxtral_tts/export_voxtral_tts.py \
    --model-path ~/models/Voxtral-4B-TTS-2603 \
    --backend cuda \
    --qlinear 4w \
    --output-dir ./voxtral_tts_exports_cuda_4w

--dtype is auto-promoted to bf16 and --qlinear-packing-format is auto-set to tile_packed_to_4d when --backend cuda --qlinear 4w is selected.

Export needs ~10 GB GPU VRAM and the full base-model checkpoint (consolidated.safetensors, ~8 GB). On RTX 5080 16 GB it takes ~6 minutes. Triton autotuning will reject some configs that exceed Blackwell's 99 KB shared-memory limit — those OutOfResources warnings are expected.

File manifest (per arch folder)

tekken.json and voice_embedding/*.pt are not in this repo — download them from `mistralai/Voxtral-4B-TTS-2603` so they always match the upstream release that this export was produced from.

Troubleshooting

• —`CUDA driver error: invalid argument` on first kernel launch. The AOTI cubins don't match your GPU's compute capability. Pick the right subfolder for your GPU, or re-export.
• —`__cudaLaunch was not declared` during build. CPATH is polluted with CUDA 13's include path. unset CPATH and rebuild.
• —`GLIBCXX_3.4.30 not found` at runner startup. AOTI .so files require a newer libstdc++ than /lib64/libstdc++.so.6. Set LD_LIBRARY_PATH=$CONDA_PREFIX/lib before launching.
• —`cannot find -lcuda` during `pip install -e .` or export (WSL2). The CUDA toolkit doesn't ship libcuda.so; on WSL2 the driver lib lives at /usr/lib/wsl/lib/. Prepend it (or /usr/local/cuda/lib64/stubs) to LIBRARY_PATH.
• —First call takes ~30–50 s. Triton autotunes the LM matmul kernels on first run, then caches per-process. The runner's warmup() absorbs this so the first user-visible synth pays the cost once.
• —`pip install -e .` after pulling source changes. The default install_executorch.sh does pip install .. Repo edits won't take effect until you reinstall as editable.

Notes

• —CUDA backend on Linux only. A Windows build would need the cuda-windows export path and a different runtime CUDA payload — see the Voxtral Realtime CUDA-Windows repo for the pattern.
• —For the in-repo guide (architecture details, all export options): the official ExecuTorch Voxtral TTS guide.

More Info

• —Official ExecuTorch Voxtral TTS guide
• —Original model
• —Sister repo: Voxtral Realtime CUDA-Windows