TechnoBaptist/higgs-audio-v3-tts-4b

1# Higgs-Audio-v3 — Audio, Voice & Expression Capabilities2 3Capabilities of `bosonai/higgs-audio-v3-tts-4b` and how they map onto this ONNX4pipeline. Source: the HF model card + the sglang-omni reference. Control tokens are5plain text tokens, so they flow through our zero-shot prompt6(`<|tts|> <|text|> tok(text) <|audio|>`) unchanged — no pipeline change needed to use them.7 8## Audio format9| Property | Value |10|---|---|11| Output | 24 kHz mono waveform (`audio_tokenizer.onnx`) |12| Frame rate | 25 fps (40 ms/frame); samples ≈ frames × 960 |13| Codec | Higgs v2 tokenizer, **8 codebooks × 1026 vocab**, delay pattern (BOC=1024, EOC=1025) |14| Backbone | Qwen3-4B, interleaved text+audio tokens |15 16## Languages17- **102 languages** total: **85 at production quality** (WER/CER < 5%) and 17 at usable18  quality (5–10%). Multilingual text is tokenized the same way; no flag needed.19 20## Voice / speaker21- **Zero-shot voice cloning** from a reference clip + reference transcript. ✅ IMPLEMENTED:22  `audio_encoder.onnx` (waveform→codes) + delay pattern + `audio_embed` splicing →23  `<|tts|> <|ref_text|> tok(ref) <|ref_audio|> [ref-code embeds] <|text|> tok(text) <|audio|>`.24  Use `--ref-audio ref.wav --ref-text "<exact transcript>"`. (Ref audio is padded to a25  multiple of 960 samples internally; clone *fidelity* needs a listen.)26- No fixed/predefined speaker slots — voice identity comes from the reference clip.27 28## Expression & emotion — inline control tokens29Syntax: **`<|category:value|>`** inserted in the text. Verified: these are real single30special tokens in the tokenizer (not split into subwords), so they're embedded and fed to31the LLM exactly as the original model expects. Example (NOTE the category is `sfx`, not32`sound_effect`): `Hello <|emotion:amusement|> that's funny <|sfx:laughter|>.`33 34- **Emotions (21):** elation, amusement, enthusiasm, determination, pride, contentment,35  affection, relief, contemplation, confusion, surprise, awe, longing, arousal, anger,36  fear, disgust, bitterness, sadness, shame, helplessness37- **Styles (3):** singing, shouting, whispering38- **Sound effects (9), category `sfx`:** `<|sfx:cough|>`, laughter, crying, screaming,39  burping, humming, sigh, sniff, sneeze (each pairs with matching onomatopoeia)40- **Environment (1):** `<|env:music|>`41- **Prosody:**42  - Speed: very_slow (~0.65×), slow (~0.85×), fast (~1.2×), very_fast (~1.4×)43  - Pitch: pitch_low (−3 semitones), pitch_high (+2.5 semitones)44  - Pauses: pause (~400–700 ms), long_pause (~700–1500 ms)45  - Delivery: expressive_high, expressive_low46 47## Using control tokens with this pipeline48They're regular text — just include them in `--text`:49```50uv run inference.py --model-path onnx/cpu_int4 \51  --text "I can't believe it <|emotion:surprise|> <|sfx:laughter|> amazing!" \52  --temperature 0.8 --top-k 50 --out expressive.wav53```54Use the EXACT token names (`emotion:`, `style:`, `sfx:`, `prosody:`, `env:music`). A55mistyped tag (e.g. `sound_effect:`) is encoded as ordinary subwords (harmless, just no56effect). Sampling (`--temperature`/`--top-k`) is recommended — pure greedy degenerates.57 58## Streaming59- **Audio-OUTPUT streaming** ✅ `stream_inference.py` — generates from a fully-known text60  prompt (standard `<|tts|>`) but decodes + emits audio in rolling chunks (left-context61  windows for clean seams), writing the wav incrementally and reporting time-to-first-chunk.62  Works for zero-shot and voice-clone. Verified end-to-end.63  `uv run stream_inference.py --model-path onnx/cpu_int4 --text "..." --chunk-frames 50 --out s.wav`64- The model's **`<|streaming_tts|>` interleaved-text-INPUT mode is NOT implemented** — the65  authoritative sglang-omni `higgs_tts` reference implements TTS only and documents NO prompt66  format for `<|streaming_tts|>` / `<|audio_cont_txt|>` / `<|await_audio|>`, so we don't67  fabricate one. (Output-streaming above covers the practical "start playback early" need.)68 69## ASR (speech→text) — NOT available, and no reference exists70- `<|asr|>` (151665) / `<|streaming_asr|>` (151666) tokens exist, but the sglang-omni71  reference does **not** implement ASR and documents no prompt format for it. Implementing it72  would be reverse-engineering (risking wrong output), AND it needs a text-logit readout73  (the tied **lm_head**, `hidden @ text_embedᵀ`) which the self-contained export doesn't ship74  as a matmul — only `text_embed` as a Gather. Wiring ASR would require both an `lm_head`/logits75  ONNX and a verified ASR prompt format. Not done.76 77## Supported summary78Zero-shot TTS, voice clone (ref audio+text), 102-language text, all inline79expression/emotion/style/sfx/prosody/env tokens, and audio-output streaming — all supported.80Not exposed: ASR, and the model's interleaved streaming-input modes.81