M2UGen/M2UGen-AudioLDM2

M²UGen Model with AudioLDM2-Music

The M²UGen model is a Music Understanding and Generation model that is capable of Music Question Answering and also Music Generation from texts, images, videos and audios, as well as Music Editing. The model utilizes encoders such as MERT for music understanding, ViT for image understanding and ViViT for video understanding and the MusicGen/AudioLDM2 model as the music generation model (music decoder), coupled with adapters and the LLaMA 2 model to make the model possible for multiple abilities.

M²UGen was published in M²UGen: Multi-modal Music Understanding and Generation with the Power of Large Language Models by Atin Sakkeer Hussain, Shansong Liu, Chenshuo Sun and Ying Shan.

The code repository for the model is published in crypto-code/M2UGen. Clone the repository, download the checkpoint and run the following for a model demo:

python gradio_app.py --model ./ckpts/M2UGen-AudioLDM2/checkpoint.pth --llama_dir ./ckpts/LLaMA-2 --music_decoder audioldm2 --music_decoder_path cvssp/audioldm2-music

Citation

If you find this model useful, please consider citing:

@article{hussain2023m,
  title={{M$^{2}$UGen: Multi-modal Music Understanding and Generation with the Power of Large Language Models}},
  author={Hussain, Atin Sakkeer and Liu, Shansong and Sun, Chenshuo and Shan, Ying},
  journal={arXiv preprint arXiv:2311.11255},
  year={2023}
}