CAMB-AI/MARS5-TTS

1---2license: agpl-3.03pipeline_tag: text-to-speech4library_name: mars5-tts5tags:6- text-to-speech7- audio8- speech9- voice-cloning10- vc11- tts12---13![MARS5 Banner](assets/github-banner.png)14 15# MARS5: A novel speech model for insane prosody.16 17This is the repo for the MARS5 English speech model (TTS) from CAMB.AI.18 19The model follows a two-stage AR-NAR pipeline with a distinctively novel NAR component (see more info in the [docs](docs/architecture.md)). 20 21With just 5 seconds of audio and a snippet of text, MARS5 can generate speech even for prosodically hard and diverse scenarios like sports commentary, anime and more. Check out our demo:22 23 24 25 26https://github.com/Camb-ai/MARS5-TTS/assets/23717819/3e191508-e03c-4ff9-9b02-d73ae0ebefdd27 28 29 30 31**Quick links**:32- [CAMB.AI website](https://camb.ai/) (access MARS5 in 140+ languages for TTS and dubbing)33- Technical docs: [in the docs folder](docs/architecture.md)34- Colab quickstart: <a target="_blank" href="https://colab.research.google.com/github/Camb-ai/mars5-tts/blob/master/mars5_demo.ipynb"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>35- Demo page with samples: [here](https://6b1a3a8e53ae.ngrok.app/)36 37![Mars 5 simplified diagram](docs/assets/simplified_diagram.png)38 39**Figure**: the high-level architecture flow of Mars 5. Given text and a reference audio, coarse (L0) encodec speech features are obtained through an autoregressive transformer model. Then, the text, reference, and coarse features are refined in a multinomial DDPM model to produce the remaining encodec codebook values. The output of the DDPM is then vocoded to produce the final audio.40 41Because the model is trained on raw audio together with byte-pair-encoded text, it can be steered with things like punctuation and capitalization.42E.g. to add a pause, add a comma to that part in the transcript. Or, to emphasize a word, put it in capital letters in the transcript. 43This enables a fairly natural way for guiding the prosody of the generated output.44 45Speaker identity is specified using an audio reference file between 2-12 seconds, with lengths around 6s giving optimal results.46Further, by providing the transcript of the reference, MARS5 enables one to do a '_deep clone_' which improves the quality of the cloning and output, at the cost of taking a bit longer to produce the audio.47For more details on this and other performance and model details, please see inside the [docs folder](docs/architecture.md).48 49 50## Quickstart51 52 53We use `torch.hub` to make loading the model easy -- no cloning of the repo needed. The steps to perform inference are simple:54 551. **Install pip dependencies**: `huggingface_hub`, `torch`, `torchaudio`, `librosa`, `vocos`, and `encodec`. Python must be at version 3.10 or greater, and torch must be v2.0 or greater.56 57```bash58pip install --upgrade torch torchaudio librosa vocos encodec huggingface_hub59```60 612. **Load models**: load the Mars 5 AR and NAR model from the huggingface hub:62 63```python64from inference import Mars5TTS, InferenceConfig as config_class65import librosa66mars5 = Mars5TTS.from_pretrained("CAMB-AI/MARS5-TTS")67# The `mars5` contains the AR and NAR model, as well as inference code.68# The `config_class` contains tunable inference config settings like temperature.69```703. **Pick a reference** and optionally its transcript:71 72```python73# load reference audio between 1-12 seconds.74wav, sr = librosa.load('<path to arbitrary 24kHz waveform>.wav', 75                       sr=mars5.sr, mono=True)76wav = torch.from_numpy(wav)77ref_transcript = "<transcript of the reference audio>"78```79 80The reference transcript is an optional piece of info you need if you wish to do a deep clone.81Mars5 supports 2 kinds of inference: a shallow, fast inference whereby you do not need the transcript of the reference (we call this a _shallow clone_), and a second slower, but typically higher quality way, which we call a _deep clone_.82To use the deep clone, you need the prompt transcript. See the [model docs](docs/architecture.md) for more info on this. 83 844. **Perform the synthesis**:85 86```python87# Pick whether you want a deep or shallow clone. Set to False if you don't know prompt transcript or want fast inference. Set to True if you know transcript and want highest quality.88deep_clone = True 89# Below you can tune other inference settings, like top_k, temperature, top_p, etc...90cfg = config_class(deep_clone=deep_clone, rep_penalty_window=100,91                      top_k=100, temperature=0.7, freq_penalty=3)92 93ar_codes, output_audio = mars5.tts("The quick brown rat.", wav, 94          ref_transcript,95          cfg=cfg)96# output_audio is (T,) shape float tensor corresponding to the 24kHz output audio.97```98 99That's it! These default settings provide pretty good results, but feel free to tune the inference settings to optimize the output for your particular example. See the [`InferenceConfig`](inference.py) code or the demo notebook for info and docs on all the different inference settings.100 101_Some tips for best quality:_102- Make sure reference audio is clean and between 1 second and 12 seconds.103- Use deep clone and provide an accurate transcript for the reference.104- Use proper punctuation -- the model can be guided and made better or worse with proper use of punctuation and capitalization.105 106 107## Model details108 109**Checkpoints**110 111The checkpoints for MARS5 are provided under the releases tab of this github repo. We provide two checkpoints:112 113- AR fp16 checkpoint [~750M parameters], along with config embedded in the checkpoint.114- NAR fp16 checkpoint [~450M parameters], along with config embedded in the checkpoint.115- The byte-pair encoding tokenizer used for the L0 encodec codes and the English text is embedded in each checkpoint under the `'vocab'` key, and follows roughly the same format of a saved minbpe tokenizer. 116 117**Hardware requirements**:118 119You must be able to store at least 750M+450M params on GPU, and do inference with 750M of active parameters. In general, at least **20GB of GPU VRAM** is needed to run the model on GPU (we plan to further optimize this in the future).120 121If you do not have the necessary hardware requirements and just want to use MARS5 in your applications, you can use it via our API: see [docs.camb.ai](https://docs.camb.ai/). If you need some more credits to test it for your use case, feel free to reach out to `help@camb.ai` for help.122 123## Roadmap124 125Mars 5 is not perfect at the moment, and we are working on a few efforts to improve its quality, stability, and performance. 126Rough areas we are looking to improve, and welcome any contributions:127 128- Improving inference stability and consistency129- Speed/performance optimizations130- Improving reference audio selection when given long references.131- Benchmark performance numbers for Mars 5 on standard speech datasets. 132 133If you would like to contribute any improvement to MARS, please feel free to contribute (guidelines below).134 135## Contributions136 137We welcome any contributions to improving the model. As you may find when experimenting, it can produce really great results, it can still be further improved to create excellent outputs _consistently_.  Please raise a PR/discussion in github.138 139**Contribution format**:140 141The preferred way to contribute to our repo is to fork the [master repository](https://github.com/Camb-ai/mars5-tts) on GitHub:142 1431. Fork the repo on github1442. Clone the repo, set upstream as this repo: `git remote add upstream git@github.com:Camb-ai/mars5-tts.git`1453. Make to a new local branch and make your changes, commit changes.1464. Push changes to new upstream branch: `git push --set-upstream origin <NAME-NEW-BRANCH>`1475. On github, go to your fork and click 'Pull request' to begin the PR process. Please make sure to include a description of what you did/fixed.148 149## License150 151We are open-sourcing MARS in English under GNU AGPL 3.0, but you can request to use it under a different license by emailing help@camb.ai152 153## Join our team154 155We're an ambitious team, globally distributed, with a singular aim of making everyone's voice count. At CAMB.AI, we're a research team of Interspeech-published, Carnegie Mellon, ex-Siri engineers and we're looking for you to join our team. 156 157We're actively hiring; please drop us an email at ack@camb.ai if you're interested. Visit our [careers page](https://www.camb.ai/careers) for more info.158 159## Acknowledgements160 161Parts of code for this project are adapted from the following repositories -- please make sure to check them out! Thank you to the authors of:162 163- AWS: For providing much needed compute resources (NVIDIA H100s) to enable training of the model.164- TransFusion: [https://github.com/RF5/transfusion-asr](https://github.com/RF5/transfusion-asr)165- Multinomial diffusion: [https://github.com/ehoogeboom/multinomial_diffusion](https://github.com/ehoogeboom/multinomial_diffusion)166- Mistral-src: [https://github.com/mistralai/mistral-src](https://github.com/mistralai/mistral-src)167- minbpe: [https://github.com/karpathy/minbpe](https://github.com/karpathy/minbpe)168- gemelo-ai's encodec Vocos: [https://github.com/gemelo-ai/vocos](https://github.com/gemelo-ai/vocos)169- librosa for their `.trim()` code: [https://librosa.org/doc/main/generated/librosa.effects.trim.html](https://librosa.org/doc/main/generated/librosa.effects.trim.html)