Pendrokar/open_tts_tracker

above models sorted by the amount of capabilities; #legend

Cloned the GitHub repo for easier viewing and embedding the above table as once requested by @reach-vb: https://github.com/Vaibhavs10/open-tts-tracker/issues/30#issuecomment-1946367525

🗣️ Open TTS Tracker

A one stop shop to track all open-access/ source Text-To-Speech (TTS) models as they come out. Feel free to make a PR for all those that aren't linked here.

This is aimed as a resource to increase awareness for these models and to make it easier for researchers, developers, and enthusiasts to stay informed about the latest advancements in the field.

[!NOTE] This repo will only track open source/access codebase TTS models. More motivation for everyone to open-source! 🤗

Some of the models are also being battle tested at TTS arenas hosted on HuggingFace:

• —🗣️ Speech Arena Leaderboard - External Speech Arena made by Artificial Analysis, who validate mostly proprietary TTS models
• —🤗🏆 TTS Spaces Arena - Mostly uses online HuggingFace Spaces, which have the Gradio API enabled
• —🏆 TTS Arena - Also has a Conversational tab that tests Podcast type of TTS
• —🎤 Expressive TTS Arena - Evaluates TTS models that "best captures the emotion, nuance, and expressiveness of human speech"
• —👥 Voice Clone Arena - Allows choosing a specific or random voice file sample for zero-shot Open TTS
• —TTS Arena v1 Archive - The above TTS-Arena-V2 was rewritten from scratch and does not use Gradio anymore

Also battle tested outside HuggingFace:

• —[Artificial Analysis](https://artificialanalysis.ai/text-to-speech) - Analysis and comparison of Text to Speech generation models & API providers. Artificial Analysis has analyzed text to speech models and hosting providers across quality, generation time, and price.

And in automated benchmarks:

• —🏅 TTSDS
• —🔥 TTSLeaderboard

• —Multilingual - Amount of supported languages is ever changing, check the Space and Hub for which specific languages are supported
• —ALL - Claims to support all natural languages; this may not include artificial/contructed languages

Also to find a model for a specific language, filter out the TTS models hosted on HuggingFace: <https://huggingface.co/models?pipeline_tag=text-to-speech&language=en&sort=trending>

Legend

For the #above TTS capability table. Open the viewer in another window or even another monitor to keep both it and the legend in view.

• —Processor ⚡ - Inference done by
• —CPU (CPUs = multithreaded) - All models can be run on CPU, so real-time factor should be below 2.0 to qualify for CPU tag, though some more leeway can be given if it supports audio streaming
• —CUDA by NVIDIA™
• —ROCm by AMD™, also see ONNX Runtime HF guide
• —Phonetic alphabet 🔤 - Phonetic transcription that allows to control pronunciation of words before inference
• —IPA - International Phonetic Alphabet
• —ARPAbet - American English focused phonetics
• —Insta-clone 👥 - Zero-shot model for quick voice cloning; Other allow only fusing multiple models or voice samples together
• —Emotion control 🎭 - Able to force an emotional state of speaker
• —🎭 <# emotions> ( 😡 anger; 😃 happiness; 😭 sadness; 😯 surprise; 🤫 whispering; 😊 friendlyness )
• —🎭👥 strict insta-clone switch - cloned on sample with specific emotion; may sound different than normal speaking voice; no ability to go in-between states
• —🎭📖 strict control through prompt - prompt input parameter
• —Prompting 📖 - Also a side effect of narrator based datasets and a way to affect the emotional state
• —📖 - Prompt as a separate input parameter
• —🗣📖 - The prompt itself is also spoken by TTS; ElevenLabs docs
• —Streaming support 🌊 - Can playback audio while it is still being generated
• —Speech control 🎚 - Ability to change the pitch, duration, etc. for the whole and/or per-phoneme of the generated speech
• —Voice conversion / Speech-To-Speech 🦜 - Streaming support implies real-time S2S; S2T=>T2S does not count
• —Longform synthesis 📜 - Able to synthesize whole paragraphs, as some TTS models tend to break down after a certain audio length limit

Example if the proprietary ElevenLabs were to be added to the capabilities table: | Name | Processor<br>⚡ | Phonetic alphabet<br>🔤 | Insta-clone<br>👥 | Emotional control<br>🎭 | Prompting<br>📖 | Speech control<br>🎚 | Streaming support<br>🌊 | Voice conversion<br>🦜 | Longform synthesis<br>📜 | |---|---|---|---|---|---|---|---|---| --- | |ElevenLabs|CUDA|IPA, ARPAbet|👥|🎭📖|🗣📖|🎚 stability, voice similarity|🌊|🦜|📜 Projects|

More info on how the capabilities table came about can be found within the GitHub Issue.

train_data Legend

Legend for the separate TTSDS Datasets (_train_data_ viewer GitHub)

• —🌐 Multilingual
• —The ISO codes of languages the model is capable off. ❌ if English only.
• —📚 Training Amount (k hours)
• —The number of hours the model was trained on
• —🧠 Num. Parameters (M)
• —How many parameters the model has, excluding vocoder and text-only components
• —🎯 Target Repr.
• —Which output representations the model uses, for example audio codecs or mel spectrograms
• —📖 LibriVox Only
• —If the model was trained on librivox-like (audiobook) data alone
• —🔄 NAR
• —If the model has a significant non-autoregressive component
• —🔁 AR
• —If the model has a significant autoregressive component
• —🔡 G2P
• —If the model uses G2P (phone inputs)
• —🧩 Language Model
• —If an LM-like approach is used (next token prediction)
• —🎵 Prosody Prediction
• —If prosodic correlates such as pitch or energy are predicted
• —🌊 Diffusion
• —If diffusion is used (outside vocoder)
• —⏱️ Delay Pattern
• —If a delay pattern is used for audio codes (see Lyth & King, 2024)

Please [create pull requests](https://huggingface.co/datasets/Pendrokar/openttstracker/edit/main/README.md) to update the info on the models within this dataset!