Modelpublic

aufklarer/Chatterbox-Flash-CoreML

sourceHugging Facemitupdated 3mo agoView on Hugging Face

0likes15kdownloads

Model Card

Chatterbox Flash CoreML

Compiled Core ML export of ResembleAI/chatterbox-flash for Apple runtimes.

This bundle contains the Chatterbox Flash T3 token generator and the S3Gen audio back-half as static-shape .mlmodelc graphs. It is intended for an application runtime that owns text tokenization, T3 denoising/sampling, reference conditioning, and graph orchestration.

Model

Component	Parameters	Format	Precision	Static shape
T3 block-diffusion token generator	532.4M	Core ML `.mlmodelc`	fp16	textlen 256, blocksize 16, max_seq 1024
S3Gen audio back-half	266.0M	Core ML `.mlmodelc`	fp16	tokenlen 192, mellen 384
Total	798.4M	Core ML `.mlmodelc`	fp16	24 kHz waveform output

Files

Path	Size	Description
`config.json`	4 KB	Root metadata for download tracking and runtime discovery
`t3/ConditioningEncoder.mlmodelc`	25 MB	Speaker/prompt/emotion conditioning to T3 conditioning embedding
`t3/TextPrefill.mlmodelc`	963 MB	Causal `[cond, text, start_speech]` prefix prefill and flat KV cache
`t3/BlockDecoder.mlmodelc`	1.0 GB	Full-visible Flash speech-block logits with explicit KV cache
`t3/uncond_block_prior.npy`	36 KB	Unconditional PMI prior for Flash scoring
`t3/tokenizer.json`	28 KB	Chatterbox Flash text tokenizer
`t3/config.json`	4 KB	T3 export metadata
`audio/FlowSpeakerProjector.mlmodelc`	44 KB	S3Gen reference embedding to projected speaker conditioning
`audio/FlowEncoder.mlmodelc`	79 MB	`prompt_token ++ speech_tokens` to flow `mu` and mask
`audio/FlowEstimator.mlmodelc`	141 MB	One meanflow Euler derivative step
`audio/HiFTVocoder.mlmodelc`	41 MB	Mel frames to 24 kHz waveform
`audio/audio_config.json`	4 KB	S3Gen audio export metadata

Runtime Boundary

The exported graphs cover:

—T3 conditioning, text prefill, and block decoding.
—S3Gen speaker projection, flow encoder, meanflow estimator step, and HiFT vocoder.

The host runtime must still provide:

—text normalization and tokenization
—T3 denoising loop, PMI/CFG scoring, sampling, and EOS trimming
—reference waveform encoders:
—VoiceEncoder: ref.wav -> speaker_emb
—prompt speech tokenizer: ref.wav -> prompt_speech_tokens
—S3Gen reference encoder: ref.wav -> prompt_token, prompt_feat, embedding
—padding/cropping and the two-step meanflow loop over (t,r) = (0,0.5), (0.5,1.0)

This means the bundle supports voice-cloning TTS when the runtime supplies reference conditioning tensors, but it is not yet a fully Core ML ref.wav -> cloned wav pipeline.

Validation

Test	Result
T3 graph roundtrip vs PyTorch wrappers	Pass at 2% relative tolerance
Audio graph roundtrip vs PyTorch wrappers	Pass for tokenlen 192, mellen 384
Stitched S3Gen meanflow-to-audio roundtrip	Pass
Prompted synthesis smoke test	Pass
Whisper tiny transcript of generated wav	`Core ML speech test.`
Smoke-test WER	0.000

Core ML warnings from local export:

—CPU_ONLY prediction crashed for the T3 package in local coremltools 8.3 testing. Use ALL, CPU_AND_NE, or a compiled-device runtime.
—The uploaded artifact ships compiled .mlmodelc folders. The numerical parity tests were run against the source .mlpackage exports before compilation.

Usage Sketch

The runtime loads graphs from t3/ and audio/, then:

1.Prepare T3 reference conditioning tensors and S3Gen ref_dict tensors from a prompt wav.
2.Tokenize text with t3/tokenizer.json.
3.Run T3 prefill and block denoising/sampling to produce S3 speech tokens.
4.Concatenate S3Gen prompt_token and generated speech tokens.
5.Run audio/FlowEncoder.mlmodelc.
6.Build cond by copying prompt_feat.T into the mel prefix.
7.Run audio/FlowEstimator.mlmodelc twice for (0,0.5) and (0.5,1.0).
8.Crop generated mel frames after the prompt prefix.
9.Run audio/HiFTVocoder.mlmodelc and crop padded samples.

Source

Converted from ResembleAI/chatterbox-flash, revision 4385507288b8197e6dab8b4e6b1603328d549d9d.