FluidInference/chatterbox-multilingual-coreml

1---2library_name: coreml3base_model:4- ResembleAI/chatterbox5license: mit6pipeline_tag: text-to-speech7tags:8- coreml9- text-to-speech10- apple-silicon11- ios12- macos13- multilingual14language:15- en16- fr17- de18- es19- it20- pt21- pl22- tr23- ru24- nl25- cs26- ar27- zh28- ja29- hu30- ko31- hi32- da33- el34- fi35- he36- ms37- no38- sv39- sw40---41 42# Chatterbox Multilingual — CoreML43 44CoreML export of [ResembleAI/chatterbox](https://huggingface.co/ResembleAI/chatterbox)45**multilingual** (23 languages, `t3_mtl23ls_v2` + `s3gen`) for Apple platforms,46converted by [FluidInference](https://github.com/FluidInference)47(conversion toolkit: [mobius PR #89](https://github.com/FluidInference/mobius/pull/89)).48 49Each model ships as both `.mlpackage` (source) and compiled `.mlmodelc`.50 51## Models52 53| File | Size (fp16) | Role | Compute |54|---|---:|---|---|55| `T3-Prefill-T256-M1024-fp16` | 977 MB | Llama-520M prefill over ≤256-token context (CFG batch 2), initializes 1024-slot KV cache | CPU+GPU |56| `T3-Decode-M1024-fp16` | 977 MB | Single-step AR decode, KV cache via I/O tensors (38 ms/step) | CPU+GPU |57| `T3-Decode-M1024-fp16-stateful` | 977 MB | Single-step AR decode, KV cache in `MLState` (**16.8 ms/step**; macOS 15+/iOS 18+) | CPU+GPU |58| `Flow-N500-fp16` | 229 MB | S3Gen flow: 500-token bucket → 1000 mel frames, 10-step CFG Euler in-graph | CPU+GPU |59| `HiFT-T1000-fp16` | 40 MB | HiFTNet vocoder: mel → 24 kHz waveform (0.09 s/call) | CPU+GPU |60| `tables/tables.safetensors` | 34 MB | text/speech embedding + learned positional tables (host applies) |61| `tables/voice-default.safetensors` | 0.1 MB | precomputed built-in voice conditioning (T3 cond embeds + S3Gen ref dict) |62| `tokenizer/grapheme_mtl_merged_expanded_v1.json` | | 23-language grapheme tokenizer |63 64⚠️ Do **not** load the T3 packages with `.cpuOnly` — prediction hard-crashes65(also independently reported by other Chatterbox CoreML ports). Use66`.cpuAndGPU` or `.all`.67 68## Samples69 70[`samples/`](./tree/main/samples) has CoreML end-to-end renders (`e2e_*.wav`)71next to stock PyTorch renders (`baseline_*.wav`) for en/de/fr, all using the72built-in voice.73 74To synthesize locally without the upstream checkpoint (Apple silicon):75 76```bash77git clone -b feat/chatterbox-mtl-coreml https://github.com/FluidInference/mobius78cd mobius/models/tts/chatterbox/coreml79uv sync80uv run python verify/e2e_coreml.py --lang en   # models auto-download from this repo81```82 83## Runtime boundary84 85The graphs cover T3 prefill/decode (with the multilingual alignment-analyzer86attention rows as outputs), the S3Gen flow, and the HiFT vocoder. The host87runtime must provide:88 89- text normalization + tokenization (`tokenizer/`)90- embedding prep from `tables.safetensors` (text/speech + positional; the91  stock prefill context ends with **two** BOS embeds — replicate exactly)92- CFG combine `cond + w*(cond-uncond)`, repetition penalty, min-p/top-p,93  sampling, EOS handling94- the `AlignmentStreamAnalyzer` heuristics, fed by the exported `align_attn`95  rows (reference port: `verify/analyzer_port.py` in the conversion toolkit)96- SineGen randomness (`phase_vec`, `noise` inputs to HiFT) and CFM noise `z`97- flow bucket padding/cropping; MLState seeding from prefill KV for the98  stateful decode99 100Voice cloning from a reference wav additionally needs the VoiceEncoder /101S3TokenizerV2 / CAMPPlus encoders, which are not converted here; voices can102be prepared offline in Python (`export-tables.py --ref-wav`) and shipped as103`voice-*.safetensors`.104 105## Parity (vs upstream PyTorch)106 107| Check | Result |108|---|---|109| T3 wrappers vs stock (fp32) | logits 3.8e-05, alignment rows exact |110| T3 CoreML fp16 | logits 2.6e-02 (range ±15), align 1.4e-03 |111| Flow CoreML fp16 | mel max 2.4e-02, mean 2.7e-03 |112| HiFT CoreML fp16 | wav max 1.7e-02, mean 2.6e-04 |113| e2e ASR round-trip (en) | exact transcript, matches PyTorch baseline |114 115## License116 117MIT, following upstream118[ResembleAI/chatterbox](https://huggingface.co/ResembleAI/chatterbox).119Upstream embeds Resemble's Perth watermarker in its Python pipeline; this120CoreML export does not include a watermarking stage.121