soapboxguy/MusicGen

1# AudioGen: Textually-guided audio generation2 3AudioCraft provides the code and a model re-implementing AudioGen, a [textually-guided audio generation][audiogen_arxiv]4model that performs text-to-sound generation.5 6The provided AudioGen reimplementation follows the LM model architecture introduced in [MusicGen][musicgen_arxiv]7and is a single stage auto-regressive Transformer model trained over a 16kHz8<a href="https://github.com/facebookresearch/encodec">EnCodec tokenizer</a> with 4 codebooks sampled at 50 Hz.9This model variant reaches similar audio quality than the original implementation introduced in the AudioGen publication10while providing faster generation speed given the smaller frame rate.11 12**Important note:** The provided models are NOT the original models used to report numbers in the13[AudioGen publication][audiogen_arxiv]. Refer to the model card to learn more about architectural changes.14 15Listen to samples from the **original AudioGen implementation** in our [sample page][audiogen_samples].16 17 18## Model Card19 20See [the model card](../model_cards/AUDIOGEN_MODEL_CARD.md).21 22 23## Installation24 25Please follow the AudioCraft installation instructions from the [README](../README.md).26 27AudioCraft requires a GPU with at least 16 GB of memory for running inference with the medium-sized models (~1.5B parameters).28 29## API and usage30 31We provide a simple API and 1 pre-trained models for AudioGen:32 33`facebook/audiogen-medium`: 1.5B model, text to sound - [🤗 Hub](https://huggingface.co/facebook/audiogen-medium)34 35You can play with AudioGen by running the jupyter notebook at [`demos/audiogen_demo.ipynb`](../demos/audiogen_demo.ipynb) locally (if you have a GPU).36 37See after a quick example for using the API.38 39```python40import torchaudio41from audiocraft.models import AudioGen42from audiocraft.data.audio import audio_write43 44model = AudioGen.get_pretrained('facebook/audiogen-medium')45model.set_generation_params(duration=5)  # generate 5 seconds.46descriptions = ['dog barking', 'sirene of an emergency vehicle', 'footsteps in a corridor']47wav = model.generate(descriptions)  # generates 3 samples.48 49for idx, one_wav in enumerate(wav):50    # Will save under {idx}.wav, with loudness normalization at -14 db LUFS.51    audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)52```53 54## Training55 56The [AudioGenSolver](../audiocraft/solvers/audiogen.py) implements the AudioGen's training pipeline57used to develop the released model. Note that this may not fully reproduce the results presented in the paper.58Similarly to MusicGen, it defines an autoregressive language modeling task over multiple streams of59discrete tokens extracted from a pre-trained EnCodec model (see [EnCodec documentation](./ENCODEC.md)60for more details on how to train such model) with dataset-specific changes for environmental sound61processing.62 63Note that **we do NOT provide any of the datasets** used for training AudioGen.64 65### Example configurations and grids66 67We provide configurations to reproduce the released models and our research.68AudioGen solvers configuration are available in [config/solver/audiogen](../config/solver/audiogen).69The base training configuration used for the released models is the following:70[`solver=audiogen/audiogen_base_16khz`](../config/solver/audiogen/audiogen_base_16khz.yaml)71 72Please find some example grids to train AudioGen at73[audiocraft/grids/audiogen](../audiocraft/grids/audiogen/).74 75```shell76# text-to-sound77dora grid audiogen.audiogen_base_16khz78```79 80### Sound dataset and metadata81 82AudioGen's underlying dataset is an AudioDataset augmented with description metadata.83The AudioGen dataset implementation expects the metadata to be available as `.json` files84at the same location as the audio files or through specified external folder.85Learn more in the [datasets section](./DATASETS.md).86 87### Evaluation stage88 89By default, evaluation stage is also computing the cross-entropy and the perplexity over the90evaluation dataset. Indeed the objective metrics used for evaluation can be costly to run91or require some extra dependencies. Please refer to the [metrics documentation](./METRICS.md)92for more details on the requirements for each metric.93 94We provide an off-the-shelf configuration to enable running the objective metrics95for audio generation in96[config/solver/audiogen/evaluation/objective_eval](../config/solver/audiogen/evaluation/objective_eval.yaml).97 98One can then activate evaluation the following way:99```shell100# using the configuration101dora run solver=audiogen/debug solver/audiogen/evaluation=objective_eval102# specifying each of the fields, e.g. to activate KL computation103dora run solver=audiogen/debug evaluate.metrics.kld=true104```105 106See [an example evaluation grid](../audiocraft/grids/audiogen/audiogen_pretrained_16khz_eval.py).107 108### Generation stage109 110The generation stage allows to generate samples conditionally and/or unconditionally and to perform111audio continuation (from a prompt). We currently support greedy sampling (argmax), sampling112from softmax with a given temperature, top-K and top-P (nucleus) sampling. The number of samples113generated and the batch size used are controlled by the `dataset.generate` configuration114while the other generation parameters are defined in `generate.lm`.115 116```shell117# control sampling parameters118dora run solver=audiogen/debug generate.lm.gen_duration=5 generate.lm.use_sampling=true generate.lm.top_k=15119```120 121## More information122 123Refer to [MusicGen's instructions](./MUSICGEN.md).124 125### Learn more126 127Learn more about AudioCraft training pipelines in the [dedicated section](./TRAINING.md).128 129 130## Citation131 132AudioGen133```134@article{kreuk2022audiogen,135    title={Audiogen: Textually guided audio generation},136    author={Kreuk, Felix and Synnaeve, Gabriel and Polyak, Adam and Singer, Uriel and D{\'e}fossez, Alexandre and Copet, Jade and Parikh, Devi and Taigman, Yaniv and Adi, Yossi},137    journal={arXiv preprint arXiv:2209.15352},138    year={2022}139}140```141 142MusicGen143```144@article{copet2023simple,145    title={Simple and Controllable Music Generation},146    author={Jade Copet and Felix Kreuk and Itai Gat and Tal Remez and David Kant and Gabriel Synnaeve and Yossi Adi and Alexandre Défossez},147    year={2023},148    journal={arXiv preprint arXiv:2306.05284},149}150```151 152## License153 154See license information in the [model card](../model_cards/AUDIOGEN_MODEL_CARD.md).155 156[audiogen_arxiv]: https://arxiv.org/abs/2209.15352157[musicgen_arxiv]: https://arxiv.org/abs/2306.05284158[audiogen_samples]: https://felixkreuk.github.io/audiogen/159