btrkeks/transcoda-59M-zeroshot-v1

Transcoda 59M Zero-Shot v1

End-to-end zero-shot Optical Music Recognition. A compact 59M-parameter vision-encoder-decoder that turns raw score images directly into `**kern` (Humdrum) symbolic transcriptions. Trained from scratch on synthetic data only.

Paired benchmarks: `btrkeks/verovio-synth-omr` (synthetic) and `btrkeks/polish-scores` (real historical scans).

Quick load (transformers)

from transformers import AutoModelForCausalLM, PreTrainedTokenizerFast

model = AutoModelForCausalLM.from_pretrained(
    "btrkeks/transcoda-59M-zeroshot-v1",
    trust_remote_code=True,
)
tokenizer = PreTrainedTokenizerFast.from_pretrained("btrkeks/transcoda-59M-zeroshot-v1")

trust_remote_code=True is required because the model uses a custom vision-encoder-decoder architecture not yet in the transformers registry. On first load the ConvNeXt-V2-Tiny base weights are re-downloaded from `facebook/convnextv2-tiny-22k-224` and then overwritten by the Transcoda checkpoint; this is expected.

Preprocessing

The model expects a single page image normalized to 1485 × 1050 pixels (height × width) with float32 pixel values in [-1, 1]:

1. 1.Convert to RGB.
2. 2.Resize to width 1050 preserving aspect ratio (bilinear).
3. 3.If the resized height exceeds 1485, top-crop; otherwise white-pad at the bottom to reach exactly 1485.
4. 4.Normalize to [-1, 1] (subtract 0.5, divide by 0.5 per channel).
5. 5.Pass as pixel_values of shape (1, 3, 1485, 1050) plus an image_sizes tensor [[1485, 1050]] to model.generate(...).

The canonical implementation is preprocess_pil_image in `src/data/preprocessing.py` in the project repository.

Full inference snippet

import torch
from PIL import Image
import numpy as np
from transformers import AutoModelForCausalLM, PreTrainedTokenizerFast

REPO = "btrkeks/transcoda-59M-zeroshot-v1"
device = "cuda" if torch.cuda.is_available() else "cpu"
model = AutoModelForCausalLM.from_pretrained(REPO, trust_remote_code=True).to(device).eval()
tokenizer = PreTrainedTokenizerFast.from_pretrained(REPO)

def preprocess(path, target_w=1050, target_h=1485):
    img = Image.open(path).convert("RGB")
    new_h = max(1, int(img.height * (target_w / img.width)))
    img = img.resize((target_w, new_h), Image.BILINEAR)
    arr = np.array(img)
    if arr.shape[0] > target_h:
        arr = arr[:target_h]
    elif arr.shape[0] < target_h:
        pad = np.full((target_h - arr.shape[0], target_w, 3), 255, dtype=arr.dtype)
        arr = np.concatenate([arr, pad], axis=0)
    t = torch.from_numpy(arr).permute(2, 0, 1).float() / 255.0
    t = (t - 0.5) / 0.5
    return t.unsqueeze(0)

pixel_values = preprocess("page.png").to(device)
image_sizes = torch.tensor([[1485, 1050]], device=device)

with torch.no_grad():
    out = model.generate(
        pixel_values=pixel_values,
        image_sizes=image_sizes,
        max_length=2048,
        do_sample=False,
        num_beams=1,
        repetition_penalty=1.1,
    )

print(tokenizer.decode(out[0], skip_special_tokens=True))

For grammar-constrained decoding (guarantees formally valid **kern) and beam search, use the project repository's scripts/inference.py.

Loading the raw .ckpt (paper reproducibility)

The original Lightning checkpoint is also distributed in this repo as transcoda-59M-zeroshot-v1.ckpt. Load it from a clone of the Transcoda repository:

import torch
from src.model.checkpoint_loader import load_model_from_checkpoint

loaded = load_model_from_checkpoint("transcoda-59M-zeroshot-v1.ckpt", torch.device("cpu"))
model = loaded.model

This matches the inference path used for the paper's reported metrics.

Architecture

Training data

310,554 synthetic page images rendered with Verovio, sourced from PDMX, Musetrainer, Grandstaff, OpenScore Lieder, and OpenScore String Quartets. Targets are canonicalized via a deterministic **kern normalization pipeline so each score maps to a single canonical sequence. Augmentation adds expressive marks (dynamics, pedals, tempo, etc.) to the rendered image without changing the target — this asymmetry teaches the decoder to ignore notation that is not part of **kern.

No real-scan or Polish data is used for training or model selection.

Benchmarks

OMR-NED on the released benchmarks (lower is better):

Headline numbers use beam search with width 3, repetition_penalty=1.1, max_length=2048. See `scripts/benchmark/README.md` in the project repository for reproduction commands and ablations.

Decoding

Three configurations are documented in the paper:

• —Greedy (default in this card's generation_config.json): do_sample=false, num_beams=1, repetition_penalty=1.1, max_length=2048.
• —Beam search (best CER, headline metric): num_beams=3, length_penalty=1.0, otherwise as above.
• —Grammar-constrained: applies a **kern grammar (`grammars/kern.gbnf`) via custom logits processors. Guarantees structural validity; requires the project repository.

The repetition penalty is preferred over no_repeat_ngram_size — music is intrinsically repetitive at every scale, and banning n-grams breaks legitimate output. See the project CLAUDE.md for details.

Limitations

• —Trained exclusively on synthetic Verovio renders. Domain gap on real scans is significant: 18.46% OMR-NED on clean Verovio pages vs 63.97% on Polish historical scans.
• —No fine-tuning on Polish or any other real-world corpus.
• —Single layout engine in training data; other engravers may degrade.
• —Fixed input geometry (1485 × 1050). Multi-page or unusual aspect ratios must be split or letterboxed by the caller.

License

CC BY 4.0.

Citation

@misc{dratschuk2026transcodaendtoendzeroshotoptical,
      title={Transcoda: End-to-End Zero-Shot Optical Music Recognition via Data-Centric Synthetic Training},
      author={Daniel Dratschuk and Paul Swoboda},
      year={2026},
      eprint={2605.10835},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2605.10835},
}