suno/bark

1---2language:3- en4- de5- es6- fr7- hi8- it9- ja10- ko11- pl12- pt13- ru14- tr15- zh16thumbnail: >-17  https://user-images.githubusercontent.com/5068315/230698495-cbb1ced9-c911-4c9a-941d-a1a4a1286ac6.png18library: bark19license: mit20tags:21- bark22- audio23- text-to-speech24pipeline_tag: text-to-speech25inference: true26---27 28# Bark29 30Bark is a transformer-based text-to-audio model created by [Suno](https://www.suno.ai). 31Bark can generate highly realistic, multilingual speech as well as other audio - including music, 32background noise and simple sound effects. The model can also produce nonverbal 33communications like laughing, sighing and crying. To support the research community, 34we are providing access to pretrained model checkpoints ready for inference.35 36The original github repo and model card can be found [here](https://github.com/suno-ai/bark).37 38This model is meant for research purposes only. 39The model output is not censored and the authors do not endorse the opinions in the generated content. 40Use at your own risk.41 42Two checkpoints are released:43- [small](https://huggingface.co/suno/bark-small)44- [**large** (this checkpoint)](https://huggingface.co/suno/bark)45 46 47## Example48 49Try out Bark yourself!50 51* Bark Colab:52 53<a target="_blank" href="https://colab.research.google.com/drive/1eJfA2XUa-mXwdMy7DoYKVYHI1iTd9Vkt?usp=sharing">54  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>55</a>56 57* Hugging Face Colab:58 59<a target="_blank" href="https://colab.research.google.com/drive/1dWWkZzvu7L9Bunq9zvD-W02RFUXoW-Pd?usp=sharing"> 60  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> 61</a>62 63* Hugging Face Demo:64 65<a target="_blank" href="https://huggingface.co/spaces/suno/bark">66  <img src="https://huggingface.co/datasets/huggingface/badges/raw/main/open-in-hf-spaces-sm.svg" alt="Open in HuggingFace"/>67</a>68 69 70## 🤗 Transformers Usage71 72You can run Bark locally with the 🤗 Transformers library from version 4.31.0 onwards.73 741. First install the 🤗 [Transformers library](https://github.com/huggingface/transformers) and scipy:75 76```77pip install --upgrade pip78pip install --upgrade transformers scipy79```80 812. Run inference via the `Text-to-Speech` (TTS) pipeline. You can infer the bark model via the TTS pipeline in just a few lines of code!82 83```python84from transformers import pipeline85import scipy86 87synthesiser = pipeline("text-to-speech", "suno/bark")88 89speech = synthesiser("Hello, my dog is cooler than you!", forward_params={"do_sample": True})90 91scipy.io.wavfile.write("bark_out.wav", rate=speech["sampling_rate"], data=speech["audio"])92```93 943. Run inference via the Transformers modelling code. You can use the processor + generate code to convert text into a mono 24 kHz speech waveform for more fine-grained control.95 96```python97from transformers import AutoProcessor, AutoModel98 99processor = AutoProcessor.from_pretrained("suno/bark")100model = AutoModel.from_pretrained("suno/bark")101 102inputs = processor(103    text=["Hello, my name is Suno. And, uh — and I like pizza. [laughs] But I also have other interests such as playing tic tac toe."],104    return_tensors="pt",105)106 107speech_values = model.generate(**inputs, do_sample=True)108```109 1104. Listen to the speech samples either in an ipynb notebook:111 112```python113from IPython.display import Audio114 115sampling_rate = model.generation_config.sample_rate116Audio(speech_values.cpu().numpy().squeeze(), rate=sampling_rate)117```118 119Or save them as a `.wav` file using a third-party library, e.g. `scipy`:120 121```python122import scipy123 124sampling_rate = model.config.sample_rate125scipy.io.wavfile.write("bark_out.wav", rate=sampling_rate, data=speech_values.cpu().numpy().squeeze())126```127 128For more details on using the Bark model for inference using the 🤗 Transformers library, refer to the [Bark docs](https://huggingface.co/docs/transformers/model_doc/bark).129 130## Suno Usage131 132You can also run Bark locally through the original [Bark library]((https://github.com/suno-ai/bark):133 1341. First install the [`bark` library](https://github.com/suno-ai/bark)135 1362. Run the following Python code:137 138```python139from bark import SAMPLE_RATE, generate_audio, preload_models140from IPython.display import Audio141 142# download and load all models143preload_models()144 145# generate audio from text146text_prompt = """147     Hello, my name is Suno. And, uh — and I like pizza. [laughs] 148     But I also have other interests such as playing tic tac toe.149"""150speech_array = generate_audio(text_prompt)151 152# play text in notebook153Audio(speech_array, rate=SAMPLE_RATE)154```155 156[pizza.webm](https://user-images.githubusercontent.com/5068315/230490503-417e688d-5115-4eee-9550-b46a2b465ee3.webm)157 158 159To save `audio_array` as a WAV file:160 161```python162from scipy.io.wavfile import write as write_wav163 164write_wav("/path/to/audio.wav", SAMPLE_RATE, audio_array)165```166 167## Model Details168 169 170The following is additional information about the models released here. 171 172Bark is a series of three transformer models that turn text into audio.173 174### Text to semantic tokens175 - Input: text, tokenized with [BERT tokenizer from Hugging Face](https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertTokenizer)176 - Output: semantic tokens that encode the audio to be generated177 178### Semantic to coarse tokens179 - Input: semantic tokens180 - Output: tokens from the first two codebooks of the [EnCodec Codec](https://github.com/facebookresearch/encodec) from facebook181 182### Coarse to fine tokens183 - Input: the first two codebooks from EnCodec184 - Output: 8 codebooks from EnCodec185 186### Architecture187|           Model           | Parameters | Attention  | Output Vocab size |  188|:-------------------------:|:----------:|------------|:-----------------:|189|  Text to semantic tokens  |    80/300 M    | Causal     |       10,000      |190| Semantic to coarse tokens |    80/300 M    | Causal     |     2x 1,024      |191|   Coarse to fine tokens   |    80/300 M    | Non-causal |     6x 1,024      |192 193 194### Release date195April 2023196 197## Broader Implications198We anticipate that this model's text to audio capabilities can be used to improve accessbility tools in a variety of languages. 199 200While we hope that this release will enable users to express their creativity and build applications that are a force201for good, we acknowledge that any text to audio model has the potential for dual use. While it is not straightforward202to voice clone known people with Bark, it can still be used for nefarious purposes. To further reduce the chances of unintended use of Bark, 203we also release a simple classifier to detect Bark-generated audio with high accuracy (see notebooks section of the main repository).