Modelpublic

FluidInference/moss-tts-nano-coreml

sourceHugging Faceapache-2.0updated 8d agoView on Hugging Face

0likes79downloads

Model Card

MOSS-TTS-Nano CoreML

On-device CoreML conversion of OpenMOSS-Team/MOSS-TTS-Nano-100M (0.1B multilingual streaming TTS with zero-shot voice cloning, 20 languages) and its codec MOSS-Audio-Tokenizer-Nano (22M, 48 kHz stereo, 12.5 Hz × 16 RVQ codebooks) for Apple platforms.

—Conversion scripts: https://github.com/FluidInference/mobius/tree/main/models/tts/moss-tts-nano/coreml
—Swift integration: https://github.com/FluidInference/FluidAudio (Sources/FluidAudio/TTS/MossTtsNano)
—Upstream: https://github.com/OpenMOSS/MOSS-TTS-Nano (Apache-2.0)

Pipeline

reference clip ─CodecEncoder─► 16×T codes ─┐
text ─SentencePiece BPE─► ids ─────────────┴─► rows [T,17] ─Prefill─► hidden + KV
                                                                  │  per 80 ms frame
                 Frame (local transformer + sampling) ─► 16 codes ─► CodecStep ─► 2×3840 samples @ 48 kHz
                 Step (global GPT-2, KV update)      ◄── [assistant_slot, 16 codes]

Files

Both .mlpackage (source) and precompiled .mlmodelc are shipped; load .mlmodelc directly.

Bundle	I/O	Target
`MossNano-Prefill-T512-M1024-fp16`	`input_ids [1,512,17] int32`, `input_len [1]` → `hidden [1,768]`, `kv_k`/`kv_v [12,1,12,1024,64]`	macOS 14 / iOS 17
`MossNano-Step-M1024-fp16`	`input_ids [1,1,17]`, `kv_k`, `kv_v`, `cur_len [1]` → `hidden`, `kv_k_out`, `kv_v_out`	macOS 14 / iOS 17
`MossNano-Frame-fp16`	`global_hidden [1,768]`, `text_u [1]`, `audio_u [1,16]`, `text_temperature`, `audio_temperature`, `audio_top_p`, `repetition_penalty` (each `[1]`), `seen [1,16,1024]`, `greedy [1]` → `should_continue [1] int32`, `frame [1,16] int32`	macOS 14 / iOS 17
`MossNano-CodecStep-fp16`	`codes [16,1,1] int32`, `frame_index [1]`, 24 KV caches `k{s}_{l}`/`v{s}_{l} [1,4,{500,800,1200,1600},64]` → `audio [1,2,3840]`, `*_out` caches	macOS 14 / iOS 17
`MossNano-CodecDecoder-fp16`	`codes [16,1,T≤125]` → `audio [1,2,T·3840]` (batch, flexible length)	macOS 14 / iOS 17
`MossNano-CodecEncoder-fp32`	`audio [1,2,S]` (S % 3840 == 0, ≤ 188 frames) → `codes [16,1,S/3840]`	macOS 14 / iOS 17
`config.json`	special token ids, pre-tokenized prompt template segments, sampling defaults
`tokenizer.model`	SentencePiece BPE (16 384 pieces, byte fallback, `nmt_nfkc`)
`voices/en_2.json`, `voices/zh_1.json`	preset voice codes (`[frames][16]`) from the upstream demo clips (also shipped as `.wav`)

Sampling runs inside the Frame graph (top-k 50 text / 25 audio fixed; temperature, top-p and repetition penalty are inputs). The host supplies uniform randoms in [0,1); greedy = 1 selects argmax. Upstream greedy decoding never emits the stop token, so use sampling (defaults: text T 1.5, audio T 1.7, top-p 0.8).

Parity and performance (M5 Pro, macOS 26.7)

—Wrappers vs upstream fp32: 1e-5 level; fp16 greedy replay of a 375-frame reference is token-exact on 370/375 frames; streaming codec step SNR 56.6 dB (GPU) vs full decode; encoder codes exact (fp32).
—Parakeet ASR on two English phrases: CoreML chain 8.3 % WER vs upstream PyTorch 10.1 %.
—Warm latency per 80 ms frame: Step 7.6 ms + Frame 5.3 ms + CodecStep 5.2 ms on GPU (≈ 4× real time streaming); Prefill 11 ms; fp32 encoder 22 ms for an 8 s prompt. Prefill/Step/CodecDecoder fail ANE compilation and should be pinned to CPU+GPU; Frame and CodecStep run on any unit.

License

Apache-2.0, following the upstream MOSS-TTS-Nano and MOSS-Audio-Tokenizer releases (see LICENSE).