soapboxguy/MusicGen

1# MusicGen: Simple and Controllable Music Generation2 3AudioCraft provides the code and models for MusicGen, [a simple and controllable model for music generation][arxiv].4MusicGen is a single stage auto-regressive Transformer model trained over a 32kHz5<a href="https://github.com/facebookresearch/encodec">EnCodec tokenizer</a> with 4 codebooks sampled at 50 Hz.6Unlike existing methods like [MusicLM](https://arxiv.org/abs/2301.11325), MusicGen doesn't require7a self-supervised semantic representation, and it generates all 4 codebooks in one pass. By introducing8a small delay between the codebooks, we show we can predict them in parallel, thus having only 50 auto-regressive9steps per second of audio.10Check out our [sample page][musicgen_samples] or test the available demo!11 12<a target="_blank" href="https://colab.research.google.com/drive/1JlTOjB-G0A2Hz3h8PK63vLZk4xdCI5QB?usp=sharing">13  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>14</a>15<a target="_blank" href="https://huggingface.co/spaces/facebook/MusicGen">16  <img src="https://huggingface.co/datasets/huggingface/badges/raw/main/open-in-hf-spaces-sm.svg" alt="Open in HugginFace"/>17</a>18<br>19 20We use 20K hours of licensed music to train MusicGen. Specifically, we rely on an internal dataset21of 10K high-quality music tracks, and on the ShutterStock and Pond5 music data.22 23 24## Model Card25 26See [the model card](../model_cards/MUSICGEN_MODEL_CARD.md).27 28 29## Installation30 31Please follow the AudioCraft installation instructions from the [README](../README.md).32 33AudioCraft requires a GPU with at least 16 GB of memory for running inference with the medium-sized models (~1.5B parameters).34 35## Usage36 37We offer a number of way to interact with MusicGen:381. A demo is also available on the [`facebook/MusicGen` Hugging Face Space](https://huggingface.co/spaces/facebook/MusicGen)39(huge thanks to all the HF team for their support).402. You can run the extended demo on a Colab:41[colab notebook](https://colab.research.google.com/drive/1JlTOjB-G0A2Hz3h8PK63vLZk4xdCI5QB?usp=sharing)423. You can use the gradio demo locally by running [`python -m demos.musicgen_app --share`](../demos/musicgen_app.py).434. You can play with MusicGen by running the jupyter notebook at [`demos/musicgen_demo.ipynb`](../demos/musicgen_demo.ipynb) locally (if you have a GPU).445. Finally, checkout [@camenduru Colab page](https://github.com/camenduru/MusicGen-colab)45which is regularly updated with contributions from @camenduru and the community.46 47 48## API49 50We provide a simple API and 10 pre-trained models. The pre trained models are:51- `facebook/musicgen-small`: 300M model, text to music only - [🤗 Hub](https://huggingface.co/facebook/musicgen-small)52- `facebook/musicgen-medium`: 1.5B model, text to music only - [🤗 Hub](https://huggingface.co/facebook/musicgen-medium)53- `facebook/musicgen-melody`: 1.5B model, text to music and text+melody to music - [🤗 Hub](https://huggingface.co/facebook/musicgen-melody)54- `facebook/musicgen-large`: 3.3B model, text to music only - [🤗 Hub](https://huggingface.co/facebook/musicgen-large)55- `facebook/musicgen-melody-large`: 3.3B model, text to music and text+melody to music - [🤗 Hub](https://huggingface.co/facebook/musicgen-melody-large)56- `facebook/musicgen-stereo-*`: All the previous models fine tuned for stereo generation -57    [small](https://huggingface.co/facebook/musicgen-stereo-small),58    [medium](https://huggingface.co/facebook/musicgen-stereo-medium),59    [large](https://huggingface.co/facebook/musicgen-stereo-large),60    [melody](https://huggingface.co/facebook/musicgen-stereo-melody),61    [melody large](https://huggingface.co/facebook/musicgen-stereo-melody-large).62 63We observe the best trade-off between quality and compute with the `facebook/musicgen-medium` or `facebook/musicgen-melody` model.64In order to use MusicGen locally **you must have a GPU**. We recommend 16GB of memory, but smaller65GPUs will be able to generate short sequences, or longer sequences with the `facebook/musicgen-small` model.66 67See after a quick example for using the API.68 69```python70import torchaudio71from audiocraft.models import MusicGen72from audiocraft.data.audio import audio_write73 74model = MusicGen.get_pretrained('facebook/musicgen-melody')75model.set_generation_params(duration=8)  # generate 8 seconds.76wav = model.generate_unconditional(4)    # generates 4 unconditional audio samples77descriptions = ['happy rock', 'energetic EDM', 'sad jazz']78wav = model.generate(descriptions)  # generates 3 samples.79 80melody, sr = torchaudio.load('./assets/bach.mp3')81# generates using the melody from the given audio and the provided descriptions.82wav = model.generate_with_chroma(descriptions, melody[None].expand(3, -1, -1), sr)83 84for idx, one_wav in enumerate(wav):85    # Will save under {idx}.wav, with loudness normalization at -14 db LUFS.86    audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)87```88 89## 🤗 Transformers Usage90 91MusicGen is available in the 🤗 Transformers library from version 4.31.0 onwards, requiring minimal dependencies92and additional packages. Steps to get started:93 941. First install the 🤗 [Transformers library](https://github.com/huggingface/transformers) from main:95 96```shell97pip install git+https://github.com/huggingface/transformers.git98```99 1002. Run the following Python code to generate text-conditional audio samples:101 102```py103from transformers import AutoProcessor, MusicgenForConditionalGeneration104 105 106processor = AutoProcessor.from_pretrained("facebook/musicgen-small")107model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")108 109inputs = processor(110    text=["80s pop track with bassy drums and synth", "90s rock song with loud guitars and heavy drums"],111    padding=True,112    return_tensors="pt",113)114 115audio_values = model.generate(**inputs, max_new_tokens=256)116```117 1183. Listen to the audio samples either in an ipynb notebook:119 120```py121from IPython.display import Audio122 123sampling_rate = model.config.audio_encoder.sampling_rate124Audio(audio_values[0].numpy(), rate=sampling_rate)125```126 127Or save them as a `.wav` file using a third-party library, e.g. `scipy`:128 129```py130import scipy131 132sampling_rate = model.config.audio_encoder.sampling_rate133scipy.io.wavfile.write("musicgen_out.wav", rate=sampling_rate, data=audio_values[0, 0].numpy())134```135 136For more details on using the MusicGen model for inference using the 🤗 Transformers library, refer to the137[MusicGen docs](https://huggingface.co/docs/transformers/main/en/model_doc/musicgen) or the hands-on138[Google Colab](https://colab.research.google.com/github/sanchit-gandhi/notebooks/blob/main/MusicGen.ipynb).139 140 141## Training142 143The [MusicGenSolver](../audiocraft/solvers/musicgen.py) implements MusicGen's training pipeline.144It defines an autoregressive language modeling task over multiple streams of discrete tokens145extracted from a pre-trained EnCodec model (see [EnCodec documentation](./ENCODEC.md)146for more details on how to train such model).147 148Note that **we do NOT provide any of the datasets** used for training MusicGen.149We provide a dummy dataset containing just a few examples for illustrative purposes.150 151Please read first the [TRAINING documentation](./TRAINING.md), in particular the Environment Setup section.152 153 154**Warning:** As of version 1.1.0, a few breaking changes were introduced. Check the [CHANGELOG.md](../CHANGELOG.md)155file for more information. You might need to retrain some of your models.156 157### Example configurations and grids158 159We provide configurations to reproduce the released models and our research.160MusicGen solvers configuration are available in [config/solver/musicgen](../config/solver/musicgen),161in particular:162* MusicGen base model for text-to-music:163[`solver=musicgen/musicgen_base_32khz`](../config/solver/musicgen/musicgen_base_32khz.yaml)164* MusicGen model with chromagram-conditioning support:165[`solver=musicgen/musicgen_melody_32khz`](../config/solver/musicgen/musicgen_melody_32khz.yaml)166 167We provide 3 different scales, e.g. `model/lm/model_scale=small` (300M), or `medium` (1.5B), and `large` (3.3B).168 169Please find some example grids to train MusicGen at170[audiocraft/grids/musicgen](../audiocraft/grids/musicgen/).171 172```shell173# text-to-music174dora grid musicgen.musicgen_base_32khz --dry_run --init175# melody-guided music generation176dora grid musicgen.musicgen_melody_base_32khz --dry_run --init177# Remove the `--dry_run --init` flags to actually schedule the jobs once everything is setup.178```179 180### Music dataset and metadata181 182MusicGen's underlying dataset is an AudioDataset augmented with music-specific metadata.183The MusicGen dataset implementation expects the metadata to be available as `.json` files184at the same location as the audio files. Learn more in the [datasets section](./DATASETS.md).185 186 187### Audio tokenizers188 189We support a number of audio tokenizers: either pretrained EnCodec models, [DAC](https://github.com/descriptinc/descript-audio-codec), or your own models.190The tokenizer is controlled with the setting `compression_model_checkpoint`.191For instance,192 193```bash194# Using the 32kHz EnCodec trained on music195dora run solver=musicgen/debug \196    compression_model_checkpoint=//pretrained/facebook/encodec_32khz \197    transformer_lm.n_q=4 transformer_lm.card=2048198 199# Using DAC200dora run solver=musicgen/debug \201    compression_model_checkpoint=//pretrained/dac_44khz \202    transformer_lm.n_q=9 transformer_lm.card=1024 \203    'codebooks_pattern.delay.delays=[0,1,2,3,4,5,6,7,8]'204 205# Using your own model after export (see ENCODEC.md)206dora run solver=musicgen/debug \207    compression_model_checkpoint=//pretrained//checkpoints/my_audio_lm/compression_state_dict.bin \208    transformer_lm.n_q=... transformer_lm.card=...209 210# Using your own model from its training checkpoint.211dora run solver=musicgen/debug \212    compression_model_checkpoint=//sig/SIG \ # where SIG is the Dora signature of the EnCodec XP.213    transformer_lm.n_q=... transformer_lm.card=...214```215 216**Warning:** you are responsible for setting the proper value for `transformer_lm.n_q` and `transformer_lm.card` (cardinality of the codebooks). You also have to update the codebook_pattern to match `n_q` as shown in the example for using DAC. .217 218 219### Training stereo models220 221Use the option `interleave_stereo_codebooks.use` set to `True` to activate stereo training along with `channels=2`. Left and right channels will be222encoded separately by the compression model, then their codebook will be interleaved, e.g. order of codebook is223`[1_L, 1_R, 2_L, 2_R, ...]`. You will also need to update the delays for the codebook patterns to match the number of codebooks, and the `n_q` value passed to the transformer LM:224```225dora run solver=musicgen/debug \226    compression_model_checkpoint=//pretrained/facebook/encodec_32khz \227    channels=2 interleave_stereo_codebooks.use=True \228    transformer_lm.n_q=8 transformer_lm.card=2048 \229    codebooks_pattern.delay.delays='[0, 0, 1, 1, 2, 2, 3, 3]'230```231 232### Fine tuning existing models233 234You can initialize your model to one of the pretrained models by using the `continue_from` argument, in particular235 236```bash237# Using pretrained MusicGen model.238dora run solver=musicgen/musicgen_base_32khz model/lm/model_scale=medium continue_from=//pretrained/facebook/musicgen-medium conditioner=text2music239 240# Using another model you already trained with a Dora signature SIG.241dora run solver=musicgen/musicgen_base_32khz model/lm/model_scale=medium continue_from=//sig/SIG conditioner=text2music242 243# Or providing manually a path244dora run solver=musicgen/musicgen_base_32khz model/lm/model_scale=medium continue_from=/checkpoints/my_other_xp/checkpoint.th245```246 247**Warning:** You are responsible for selecting the other parameters accordingly, in a way that make it compatible248    with the model you are fine tuning. Configuration is NOT automatically inherited from the model you continue from. In particular make sure to select the proper `conditioner` and `model/lm/model_scale`.249 250**Warning:** We currently do not support fine tuning a model with slightly different layers. If you decide251 to change some parts, like the conditioning or some other parts of the model, you are responsible for manually crafting a checkpoint file from which we can safely run `load_state_dict`.252 If you decide to do so, make sure your checkpoint is saved with `torch.save` and contains a dict253    `{'best_state': {'model': model_state_dict_here}}`. Directly give the path to `continue_from` without a `//pretrained/` prefix.254 255 256#### Fine tuning mono model to stereo257 258You will not be able to `continue_from` a mono model with stereo training, as the shape of the embeddings and output linears259would not match. You can use the following snippet to prepare a proper finetuning checkpoint.260 261```python262from pathlib import Path263import torch264 265# Download the pretrained model, e.g. from266# https://huggingface.co/facebook/musicgen-melody/blob/main/state_dict.bin267 268model_name = 'musicgen-melody'269root = Path.home() / 'checkpoints'270# You are responsible for downloading the following checkpoint in the proper location271input_state_dict_path = root / model_name / 'state_dict.bin'272state = torch.load(input_state_dict_path, 'cpu')273bs = state['best_state']274# there is a slight different in format between training checkpoints and exported public checkpoints.275# If you want to use your own mono models from one of your training checkpont, following the instructions276# for exporting a model explained later on this page.277assert 'model' not in bs, 'The following code is for using an exported pretrained model'278nbs = dict(bs)279for k in range(8):280    # We will just copy mono embeddings and linears twice, once for left and right channels.281    nbs[f'linears.{k}.weight'] = bs[f'linears.{k//2}.weight']282    nbs[f'emb.{k}.weight'] = bs[f'emb.{k//2}.weight']283torch.save({'best_state': {'model': nbs}}, root / f'stereo_finetune_{model_name}.th')284```285 286Now, you can use `$HOME/checkpoints/stereo_finetune_musicgen-melody.th` as a `continue_from` target (without a `//pretrained` prefix!).287 288### Caching of EnCodec tokens289 290It is possible to precompute the EnCodec tokens and other metadata.291An example of generating and using this cache provided in the [musicgen.musicgen_base_cached_32khz grid](../audiocraft/grids/musicgen/musicgen_base_cached_32khz.py).292 293### Evaluation stage294 295By default, evaluation stage is also computing the cross-entropy and the perplexity over the296evaluation dataset. Indeed the objective metrics used for evaluation can be costly to run297or require some extra dependencies. Please refer to the [metrics documentation](./METRICS.md)298for more details on the requirements for each metric.299 300We provide an off-the-shelf configuration to enable running the objective metrics301for audio generation in302[config/solver/musicgen/evaluation/objective_eval](../config/solver/musicgen/evaluation/objective_eval.yaml).303 304One can then activate evaluation the following way:305```shell306# using the configuration307dora run solver=musicgen/debug solver/musicgen/evaluation=objective_eval308# specifying each of the fields, e.g. to activate KL computation309dora run solver=musicgen/debug evaluate.metrics.kld=true310```311 312See [an example evaluation grid](../audiocraft/grids/musicgen/musicgen_pretrained_32khz_eval.py).313 314### Generation stage315 316The generation stage allows to generate samples conditionally and/or unconditionally and to perform317audio continuation (from a prompt). We currently support greedy sampling (argmax), sampling318from softmax with a given temperature, top-K and top-P (nucleus) sampling. The number of samples319generated and the batch size used are controlled by the `dataset.generate` configuration320while the other generation parameters are defined in `generate.lm`.321 322```shell323# control sampling parameters324dora run solver=musicgen/debug generate.lm.gen_duration=10 generate.lm.use_sampling=true generate.lm.top_k=15325```326 327#### Listening to samples328 329Note that generation happens automatically every 25 epochs. You can easily access and330compare samples between models (as long as they are trained) on the same dataset using the331MOS tool. For that first `pip install Flask gunicorn`. Then332```333gunicorn -w 4 -b 127.0.0.1:8895 -t 120 'scripts.mos:app'  --access-logfile -334```335And access the tool at [https://127.0.0.1:8895](https://127.0.0.1:8895).336 337### Playing with the model338 339Once you have launched some experiments, you can easily get access340to the Solver with the latest trained model using the following snippet.341 342```python343from audiocraft.solvers.musicgen import MusicGen344 345solver = MusicGen.get_eval_solver_from_sig('SIG', device='cpu', batch_size=8)346solver.model347solver.dataloaders348```349 350### Importing / Exporting models351 352We do not support currently loading a model from the Hugging Face implementation or exporting to it.353If you want to export your model in a way that is compatible with `audiocraft.models.MusicGen`354API, you can run:355 356```python357from audiocraft.utils import export358from audiocraft import train359xp = train.main.get_xp_from_sig('SIG_OF_LM')360export.export_lm(xp.folder / 'checkpoint.th', '/checkpoints/my_audio_lm/state_dict.bin')361# You also need to bundle the EnCodec model you used !!362## Case 1) you trained your own363xp_encodec = train.main.get_xp_from_sig('SIG_OF_ENCODEC')364export.export_encodec(xp_encodec.folder / 'checkpoint.th', '/checkpoints/my_audio_lm/compression_state_dict.bin')365## Case 2) you used a pretrained model. Give the name you used without the //pretrained/ prefix.366## This will actually not dump the actual model, simply a pointer to the right model to download.367export.export_pretrained_compression_model('facebook/encodec_32khz', '/checkpoints/my_audio_lm/compression_state_dict.bin')368```369 370Now you can load your custom model with:371```python372import audiocraft.models373musicgen = audiocraft.models.MusicGen.get_pretrained('/checkpoints/my_audio_lm/')374```375 376 377### Learn more378 379Learn more about AudioCraft training pipelines in the [dedicated section](./TRAINING.md).380 381## FAQ382 383#### I need help on Windows384 385@FurkanGozukara made a complete tutorial for [AudioCraft/MusicGen on Windows](https://youtu.be/v-YpvPkhdO4)386 387#### I need help for running the demo on Colab388 389Check [@camenduru tutorial on YouTube](https://www.youtube.com/watch?v=EGfxuTy9Eeo).390 391#### What are top-k, top-p, temperature and classifier-free guidance?392 393Check out [@FurkanGozukara tutorial](https://github.com/FurkanGozukara/Stable-Diffusion/blob/main/Tutorials/AI-Music-Generation-Audiocraft-Tutorial.md#more-info-about-top-k-top-p-temperature-and-classifier-free-guidance-from-chatgpt).394 395#### Should I use FSDP or autocast ?396 397The two are mutually exclusive (because FSDP does autocast on its own).398You can use autocast up to 1.5B (medium), if you have enough RAM on your GPU.399FSDP makes everything more complex but will free up some memory for the actual400activations by sharding the optimizer state.401 402## Citation403```404@article{copet2023simple,405    title={Simple and Controllable Music Generation},406    author={Jade Copet and Felix Kreuk and Itai Gat and Tal Remez and David Kant and Gabriel Synnaeve and Yossi Adi and Alexandre Défossez},407    year={2023},408    journal={arXiv preprint arXiv:2306.05284},409}410```411 412 413## License414 415See license information in the [model card](../model_cards/MUSICGEN_MODEL_CARD.md).416 417 418[arxiv]: https://arxiv.org/abs/2306.05284419[musicgen_samples]: https://ai.honu.io/papers/musicgen/420