CoolFace
Modelpublic

csc-unipd/lilybert

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face
1likes45downloads
Model Card

lilyBERT

lilyBERT is a masked language model for LilyPond music notation, built by adapting CodeBERT to the musical domain.

LilyPond is a text-based music engraving language with formal grammar, block structure, and backslash commands — making it structurally similar to a programming language. lilyBERT leverages this by extending CodeBERT's vocabulary with 115 domain-specific tokens (e.g. \trill, \fermata, \mordent, \staccato) and performing MLM pre-training on curated Baroque music scores.

Training

This checkpoint was trained in two stages:

  1. 1.Stage 1 — PDMX pre-training: CodeBERT fine-tuned on the PDMX corpus of automatically converted LilyPond files.
  2. 2.Stage 2 — BMdataset fine-tuning: Further fine-tuned on the BMdataset, a musicologically curated collection of ~470 Baroque scores in LilyPond format (~90M tokens).
HyperparameterValue
ArchitectureRobertaForMaskedLM (12 layers, 768 hidden, 12 heads)
Vocab size50,380 (50,265 base + 115 music tokens)
Max sequence length512
MLM probability0.15
Batch size72 × 2 GPUs × 2 grad. accum. = 288
Learning rate2e-4 (cosine schedule)
Warmup10%
Epochs10 (early stopping, patience 5)
Precisionbf16
OptimizerAdamW (fused)

Results

Linear probing on the out-of-domain Mutopia corpus (layer 6, 5-fold CV):

ModelComposer Acc.Style Acc.
CB + PDMX_full (15B tokens)80.882.6
CB + BMdataset (90M tokens)82.983.7
CB + PDMX_90M (90M tokens)81.782.3
CB + PDMX → BM (this model)84.382.9

90M tokens of expertly curated data outperform 15B tokens of automatically converted data. Combining broad pre-training with domain-specific fine-tuning yields the best overall composer accuracy.

Usage

python
from transformers import AutoModelForMaskedLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("csc-unipd/lilybert")
model = AutoModelForMaskedLM.from_pretrained("csc-unipd/lilybert")

Fill-mask example

python
from transformers import pipeline

filler = pipeline("fill-mask", model="csc-unipd/lilybert")
filler("\\relative c' { c4 d <mask> f | g2 g }")

Feature extraction

python
import torch

inputs = tokenizer("\\relative c' { c4 d e f | g2 g }", return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs, output_hidden_states=True)

# Layer 6 embeddings (best for linear probing)
embeddings = outputs.hidden_states[6]

Citation

bibtex
@misc{spanio2026lilybert,
      title={BMdataset: A Musicologically Curated LilyPond Dataset}, 
      author={Matteo Spanio and Ilay Guler and Antonio Rodà},
      year={2026},
      eprint={2604.10628},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2604.10628}, 
}
bibtex
@misc{spanio2026llmsunderstandlilypondbenchmark,
      title={Can LLMs understand LilyPond? A benchmark for symbolic music generation and understanding}, 
      author={Matteo Spanio and Mohammad Torabi and Andrea Poltronieri and Antonio Rodà},
      year={2026},
      eprint={2606.08722},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2606.08722}, 
}

Links

License

Apache-2.0