owaski/moss-tts-realtime-delta-zh-125k

MOSS-TTS-Realtime, finetuned for delta-by-delta Chinese speech

The speech half of a cascaded simultaneous speech-to-speech system. A simultaneous translation model emits short Chinese text deltas, one per input chunk; this model speaks each delta as it arrives, continuing the same utterance rather than starting a new one.

Internal run id: moss_sft_train_xl125k_zh_origin_10231704 (checkpoint-epoch-0).

Why it is finetuned at all

Stock MOSS-TTS-Realtime is asked here to do something it was not trained for: speak a two-to-six-character fragment, then another, then another, and have the result sound like one sentence. Finetuning on a corpus cut the same way the server feeds it — one turn per emission step, cut from a single synthesized passage — is what closes that gap. Against the stock model at the same condition it is worth roughly 13 s of LongYAAL (20596 → 7153 ms CA at 1920 ms chunks).

Measured results

ACL 60/60 dev, en→zh, 1.0× speed, chunk 1920 ms, paired with `owaski/infinisst-thinker-phrase-zh`. Generation job 10279835.

The BLEU is a property of the pair, not of this model alone — the TTS is scored by transcribing what it says. What this checkpoint moves is latency and intelligibility of short deltas: 6.0% of ≤4-character deltas come out near-silent, against 18.6% for a variant trained on a phrase-gated corpus.

Training

• —Base: OpenMOSS-Team/MOSS-TTS-Realtime; codec OpenMOSS-Team/MOSS-Audio-Tokenizer
• —Data: 125k-row en→zh TTS target corpus built from GigaSpeech — each row one utterance synthesized as a whole, then cut at word-alignment boundaries into per-delta turns
• —Recipe: 1 epoch, bf16, lr 1e-5, linear schedule with 3% warmup, global batch 16 (1 × 4 GPUs × 4 accumulation), 7,733 optimizer steps, weight decay 0.1, grad clip 1.0, seed 42
• —Full arguments in finetune_args.json in this repo.

Use

Served behind a small HTTP shim and driven one delta at a time, with the codec context carried across turns (--codec-context conversation) — resetting the codec stream per turn is audible as voice drift. The speaker prompt is a fixed zero-shot reference clip; keep it fixed across runs you intend to compare, since the prompt changes synthesized duration and therefore every latency number.

mossttsrealtime/ ships the modeling code this checkpoint was trained and served with, including a locally modified modeling_mossttsrealtime_local.py.

Caveats

• —Chinese output only, driven by English-source deltas.
• —Trained on word-aligned turn boundaries. It is the right pairing for a word-aligned or phrase-gated thinker, but a TTS corpus that is itself phrase-gated performs worse — see the thinker card.
• —One dev split at one chunk size; other chunk sizes are different operating points.