slseanwu/MIDI-LLM_Llama-3.2-1B

MIDI-LLM

Built on Llama 3.2 (1B) with an extended vocabulary for MIDI tokens.

Research Paper

• —Shih-Lun Wu, Dave Carlton, Ryan Mikayawa, Yoon Kim, Chris Donahue, and Cheng-Zhi Anna Huang. "MIDI-LLM: Improving text-to-MIDI music generation via adapting large language models." International Society for Music Information Retrieval Conference (ISMIR), 2026. MIT, Hooktheory, CMU [Code] [Live Demo] [Paper] [Video]

Model Description

• —Base Model: meta-llama/Llama-3.2-1B
• —Model Size: 1.4B parameters
• —Extended Vocabulary: 183,286 tokens (128,256 for text + 55,030 for MIDI music)
• —Architecture: LlamaForCausalLM with extended embedding layer
• —Precision: BFloat16

Quick Start

Clone our Github code repo, run through setup steps, and try:

git clone https://github.com/slSeanWU/MIDI-LLM
cd MIDI-LLM

python generate_transformers.py \
    --model slseanwu/MIDI-LLM_Llama-3.2-1B \
    --prompt "A cheerful rock song with bright electric guitars" \
    --n_outputs 4

The repo and inference scripts provide a more complete usage guide.

Model Details

Extended Vocabulary

The model extends Llama 3.2's vocabulary (128,256 tokens) with 55,030 MIDI tokens representing:

• —Onset time (when notes occur)
• —Durations (how long each note is held)
• —Instrument-pitch pair (which note to play & by which instrument)

These tokens follow the vocabulary of Anticipatory Music Transformer (AMT) (Thickstun et al., TMLR 2024).

Training Data

• —Datasets:
• —Continued Pretraining (CPT)
• —music-related text from [MusicPile](https://huggingface.co/datasets/m-a-p/MusicPile) (~1.7B tokens)
• —standalone MIDIs from [GigaMIDI](https://huggingface.co/datasets/Metacreation/GigaMIDI) (~1.4B tokens after filtering out SFT examples)
• —Supervised Finetuning (SFT)
• —[LakhMIDI](https://colinraffel.com/projects/lmd/) music paired w/ [MidiCaps](https://huggingface.co/datasets/amaai-lab/MidiCaps) text descriptions (~5B tokens with AMT infilling augmentation)
• —Training objective: Causal language modeling
• —Training sequence length: 2,048
• —System prompt:

### Inference Hyperparameters

Recommended settings for best results:

temperature: 1.0 topp: 0.98 maxtokens: 2046

## Evaluation
This model checkpoint was evaluated with **FAD** and **CLAP** metrics on 896 LakhMIDI examples whose IDs can be found in our repo
- https://github.com/slSeanWU/MIDI-LLM/blob/main/assets/evaluation_set_lakh_ids.txt
| Model | Params | Precision | FAD ↓ | CLAP ↑ |
|-------|--------|-----------|-------|--------|
| **MIDI-LLM** | 1.47B | BF16 | **0.173** | **22.1** |
| **MIDI-LLM** | 1.47B | FP8 | 0.216 | 21.8 |


## Citation

If you find our model useful, please cite our research as

@inproceedings{wu2026midillm, title={{MIDI-LLM}: Improving text-to-{MIDI} music generation via adapting large language models}, author={Wu, Shih-Lun and Carlton, Dave and Mikayawa, Ryan and Kim, Yoon and Donahue, Chris and Huang, Cheng-Zhi Anna}, booktitle={Proc. Int. Soc. for Music Information Retrieval Conference (ISMIR)}, year={2026} }

## License

This model is based on Llama 3.2 and is subject to the [Llama 3.2 Community License](https://www.llama.com/llama3_2/license/).