OpenMOSS-Team/MOSS-Music-8B-Instruct
382.7k
1---2license: apache-2.03language:4- en5- zh6library_name: transformers7pipeline_tag: audio-text-to-text8tags:9- music10- music-understanding11- audio12- audio-language-model13- moss14- moss-music15- lyrics-asr16- music-captioning17- chord-recognition18---19# MOSS-Music20 21<p align="center">22 <img src="./assets/MOSS-Music.png" width="58%" alt="MOSS-Music logo" />23</p>24 25<div align="center">26 27<a href="https://github.com/OpenMOSS/MOSS-Music"><img src="https://img.shields.io/badge/GitHub-Repo-black?logo=github&"></a>28<img src="https://img.shields.io/badge/Blog-Coming_Soon-blue?logo=internet-explorer&">29<img src="https://img.shields.io/badge/Arxiv-Coming_Soon-red?logo=Arxiv&">30 31<a href="https://x.com/Open_MOSS"><img src="https://img.shields.io/badge/Twitter-Follow-black?logo=x&"></a>32<a href="https://discord.gg/Xf3aXddCjc"><img src="https://img.shields.io/badge/Discord-Join-5865F2?logo=discord&"></a>33 34</div>35 36<p align="center">37 <a href="./README.md">English</a> | <a href="./README_zh.md">简体中文</a>38</p>39 40**MOSS-Music** is an open-source **music understanding model** from41[MOSI.AI](https://mosi.cn/#hero), the [OpenMOSS team](https://www.open-moss.com/),42and [Shanghai Innovation Institute](https://www.sii.edu.cn/). Built on the same43audio backbone as [MOSS-Audio](https://github.com/OpenMOSS/MOSS-Audio),44MOSS-Music is further specialised on music via dedicated continual pre-training45and supervised fine-tuning — targeting **musical captioning, lyrics ASR,46structural analysis, chord / key / tempo reasoning, and long-form musical47question answering**. In this release, we provide **two 8B models**:48**MOSS-Music-8B-Instruct** and **MOSS-Music-8B-Thinking**. The Instruct variant49is optimised for direct instruction following on musical prompts, while the50Thinking variant provides stronger chain-of-thought reasoning for musical51analysis.52 53## News54 55* 2026.05.01: 🎉🎉🎉 We have released [MOSS-Music](https://github.com/OpenMOSS/MOSS-Music).56* 2026.05.01: 🎉🎉🎉 We have released [MOSS-Music-Data-Pipeline](https://github.com/wx9songs/MOSS-Music-Data-Pipeline) for large-scale music data annotation and processing.57 58## Contents59 60- [Introduction](#introduction)61- [Model Architecture](#model-architecture)62 - [DeepStack Cross-Layer Feature Injection](#deepstack-cross-layer-feature-injection)63 - [Time-Aware Representation](#time-aware-representation)64- [Released Models](#released-models)65- [Music Data Pipeline](#music-data-pipeline)66- [Evaluation](#evaluation)67- [Quickstart](#quickstart)68 - [Environment Setup](#environment-setup)69 - [SGLang Serving](#sglang-serving)70 - [Gradio App](#gradio-app)71- [More Information](#more-information)72- [LICENSE](#license)73- [Citation](#citation)74 75## Introduction76 77Music is not just audio plus lyrics — understanding it requires perceiving78harmonic structure, rhythm, timbre, instrumentation, performance nuance, and79the textual content of the lyrics, and reasoning about them jointly across80time. **MOSS-Music** is built to unify these capabilities within a single81model.82 83- **Lyrics ASR & time-aligned transcription**: Accurate singing ASR with84 sentence- and word-level timestamps, robust to backing tracks.85- **Musical captioning & tagging**: Natural-language descriptions of mood,86 genre, instrumentation, production style, and emotional trajectory.87- **Key / tempo / chord reasoning**: Identifies musical key, beats, downbeats,88 and chord progressions, including timestamped chord transcription.89- **Structural analysis**: Segments a song into intro / verse / chorus /90 bridge / outro and reasons about repetition and contrast.91- **Instrument & voice recognition**: Identifies prominent instruments and92 singing voices (solo / chorus, gender, register).93- **Musical QA and long-form analysis**: Open-ended question answering94 grounded in a full track, including chain-of-thought reasoning in the95 *Thinking* variant.96 97<p align="center">98 <img src="./assets/moss-music_img.png" width="98%" alt="MOSS-Music overview" />99</p>100 101## Model Architecture102 103MOSS-Music inherits the MOSS-Audio modular design, comprising three104components: an audio encoder, a modality adapter, and a large language model.105Raw audio is first encoded by **MOSS-Audio-Encoder** into continuous temporal106representations at **12.5 Hz**, which are then projected into the language107model's embedding space through the adapter and finally consumed by the LLM108for auto-regressive text generation.109 110Rather than relying on off-the-shelf audio frontends, we train a dedicated111encoder from scratch to obtain more robust acoustic representations, tighter112temporal alignment, and better extensibility across musical styles, singing,113and non-speech acoustic content.114 115### DeepStack Cross-Layer Feature Injection116 117Using only the encoder's top-layer features tends to lose low-level prosody,118transient events, and local time-frequency structure. To address this, we119adopt a **DeepStack**-inspired cross-layer injection module between the120encoder and the language model: in addition to the encoder's final-layer121output, features from earlier and intermediate layers are selected,122independently projected, and injected into the language model's early layers,123preserving multi-granularity information from low-level acoustic details to124high-level semantic abstractions.125 126This design is especially well-suited for music understanding, as it helps127retain rhythm, timbre, transients, and instrumental texture — information128that a single high-level representation cannot fully capture, yet is critical129for chord recognition, structural analysis, and nuanced musical description.130 131### Time-Aware Representation132 133Time is a critical dimension in music understanding. To enhance explicit134temporal awareness, we adopt a **time-marker insertion** strategy during135pre-training: explicit time tokens are inserted between audio frame136representations at fixed time intervals to indicate temporal positions.137This design enables the model to learn "what happened when" within a unified138text generation framework, naturally supporting timestamped lyrics ASR,139beat / downbeat localisation, section boundary detection, and long-song140retrospective QA.141 142Building on the MOSS-Audio backbone, MOSS-Music is further enhanced through:143 144- **continual pre-training** on a large, diverse music corpus produced by145 the data annotation and processing pipeline146 [`MOSS-Music-Data-Pipeline`](https://github.com/wx9songs/MOSS-Music-Data-Pipeline),147 with an emphasis on singing, lyrics, and full-song coverage;148- **supervised fine-tuning (SFT)** on music-centric instruction data covering149 captioning, lyrics ASR, chord / key / structural analysis, and long-form150 musical QA;151- additional **reasoning tuning** for the *Thinking* variant.152 153## Released Models154 155| Model | Audio Encoder | LLM Backbone | Total Size | Hugging Face | ModelScope |156|---|---|---|---:|---|---|157| **MOSS‑Music‑8B‑Instruct** | MOSS-Audio-Encoder | Qwen3-8B | ~9.1B | [](https://huggingface.co/OpenMOSS-Team/MOSS-Music-8B-Instruct) | [](https://modelscope.cn/models/openmoss/MOSS-Music-8B-Instruct) |158| **MOSS‑Music‑8B‑Thinking** | MOSS-Audio-Encoder | Qwen3-8B | ~9.1B | [](https://huggingface.co/OpenMOSS-Team/MOSS-Music-8B-Thinking) | [](https://modelscope.cn/models/openmoss/MOSS-Music-8B-Thinking) |159 160> Smaller (4B) variants and additional sizes may follow. Stay tuned!161 162## Music Data Pipeline163 164The training data used by MOSS-Music is produced by an end-to-end pipeline165that goes from raw audio to chat-formatted training samples. That pipeline is166available at167[`MOSS-Music-Data-Pipeline`](https://github.com/wx9songs/MOSS-Music-Data-Pipeline),168which hosts duration detection, MIR feature extraction, song-structure169segmentation, lyrics ASR, metadata cleanup, and ALM-driven caption / query170generation with models such as Qwen3-Omni, MusicFlamingo, and other171audio-language models.172 173<p align="center">174 <img src="./assets/music_pipeline.png" width="94%" />175</p>176 177## Evaluation178 179We evaluate MOSS-Music on a diverse suite of public music understanding180benchmarks. Key results:181 182- **Music QA and understanding**: **MOSS-Music-8B-Instruct** achieves **80.38**183 average accuracy across **8 public music QA benchmarks** (excluding the184 three NSynth note-recognition tracks), ranking first among all compared185 models in our current evaluation set.186- **Music captioning**: In our preliminary **GPT-5.4-as-a-Judge** evaluation,187 the MOSS-Music series leads both caption benchmarks, with188 `MOSS-Music-8B-Thinking` reaching **4.53** on `MusicCaps` and189 `MOSS-Music-8B-Instruct` reaching **4.58** on `SDD`.190- **Lyrics ASR for singing voice**: **MOSS-Music-8B-Thinking** achieves the191 best average lyrics recognition error across `MUSDB18`, `MIR-1K` and192 `Opencpop` (**15.88%** avg WER/CER), clearly ahead of all compared193 audio-language baselines including `Gemini-3.1-Pro-Preview`,194 `MusicFlamingo` and `Qwen3-Omni`. Detailed timestamped-ASR results will be195 released in a later update.196- **Chord transcription**: MOSS-Music supports chord transcription, including197 timestamped chord transcription for harmonic analysis, accompaniment198 reference, and related downstream use cases. Detailed benchmark results will199 be released in a later update.200 201 202<p align="center">203 <img src="./assets/music_bench.png" width="98%" />204</p>205 206### Music QA & Understanding (Accuracy↑)207 208| Model | MMAU-music | MMAU-mini-music | MMAU-Pro-music | MMAR-music | MuChoMusic | Music-AVQA | NSynth (instrument) | NSynth (source) | NSynth (pitch) | GTZAN | Medley-Solos-DB | Avg |209|-----|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|210| **MOSS‑Music‑8B‑Instruct** | **79.33** | **80.78** | 71.02 | 59.70 | **89.39** | **76.78** | **86.55** | 61.07 | **86.94** | **93.59** | 92.42 | **80.38** |211| Gemini‑3.1‑Pro | 71.69 | 77.18 | **73.06** | **71.64** | 79.53 | 61.51 | 13.38 | 38.90 | 6.47 | 86.39 | 80.34 | 75.17 |212| **MOSS‑Music‑8B‑Thinking** | 74.09 | 77.78 | 67.98 | 50.25 | 82.90 | 68.90 | 56.17 | 57.48 | 77.83 | 84.78 | 87.42 | 74.26 |213| MusicFlamingo | 76.83 | 76.35 | 65.60 | 48.66 | 74.58 | 73.60 | 80.76 | **75.89** | 0.00 | 84.45 | 90.86 | 73.87 |214| Audio‑Flamingo‑Next | 72.39 | 72.07 | 61.64 | 45.27 | 75.62 | 62.94 | 86.40 | 66.73 | 0.05 | 77.68 | 91.47 | 69.89 |215| MiMo‑Audio‑7B‑Instruct | 66.36 | 72.97 | 66.50 | 45.77 | 75.40 | 57.05 | 25.01 | 1.49 | 4.86 | 65.67 | **93.81** | 67.94 |216| Step‑Audio‑R1 | 66.46 | 75.08 | 62.34 | 50.75 | 72.62 | 57.98 | 13.75 | 15.87 | 2.39 | 73.67 | 82.45 | 67.67 |217| Qwen3‑Omni | 65.76 | 68.77 | 66.27 | 48.54 | 78.77 | 56.05 | 30.92 | 44.30 | 28.08 | 80.15 | 69.65 | 66.75 |218| Kimi‑Audio‑7B‑Instruct | 47.95 | 52.25 | 59.10 | 45.27 | 70.18 | 68.90 | 6.01 | 0.81 | 3.88 | 39.54 | 71.98 | 56.90 |219 220> `Avg` is computed over 8 public music QA benchmarks:221> `MMAU-music`, `MMAU-mini-music`, `MMAU-Pro-music`, `MMAR-music`,222> `MuChoMusic`, `Music-AVQA`, `GTZAN`, and `Medley-Solos-DB`.223>224> We exclude the three `NSynth` tracks from the main average because they focus225> on fine-grained isolated-note recognition, including instrument-family,226> acoustic/electronic source, and exact pitch discrimination from short227> single-note clips. Some compared audio-language models are not explicitly228> designed for this note-level classification setting, so we report NSynth229> separately for reference rather than mixing it into the headline average.230 231### Music Captioning232 233We further report a preliminary **GPT-5.4-as-a-Judge** music captioning234comparison on `MusicCaps` and `Song Describer Dataset (SDD)`. Scores are on a2351-5 scale across 9 dimensions: `genre/style`, `mood/affect`, `tempo/rhythm`,236`instrumentation/timbre`, `vocals`, `melody/harmony`, `structure/form`,237`production/audio quality`, and `scene/use case`.238 239- **Overall caption quality**: the MOSS-Music series remains strongest across240 both caption benchmarks, with `MOSS-Music-8B-Thinking` reaching **4.53** on241 `MusicCaps` and `MOSS-Music-8B-Instruct` reaching **4.58** on `SDD`.242- **Stronger structural descriptions**: MOSS-Music shows the clearest gains on243 `structure / form / progression`, especially on `SDD`.244- **Competitive baselines on instrumentation and scene semantics**:245 `MusicFlamingo` and `Gemini-3.1-Pro` remain competitive on246 `instrumentation/timbre`, while `Gemini-3.1-Pro` is strongest on247 `scene / use case`.248 249#### MusicCaps250 251| Model | Genre | Mood | Tempo | Instr. | Vocals | Melody/Harmony | Structure | Production | Scene | Avg |252|-----|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|253| **MOSS‑Music‑8B‑Thinking** | 4.78 | **4.69** | **4.62** | 4.40 | **4.46** | **4.40** | **4.86** | 4.35 | 4.18 | **4.53** |254| Gemini‑3.1‑Pro | 4.70 | 4.60 | 4.48 | **4.68** | 4.18 | 4.18 | 3.86 | **4.40** | **4.72** | 4.42 |255| **MOSS‑Music‑8B‑Instruct** | 4.60 | 4.52 | 4.46 | 4.02 | 4.30 | 4.38 | 4.78 | 4.20 | 3.96 | 4.36 |256| MusicFlamingo | **4.80** | 4.36 | 4.50 | 4.64 | 3.94 | 4.08 | 3.58 | 4.30 | 3.72 | 4.21 |257| Audio‑Flamingo‑Next | 4.34 | 4.56 | 4.08 | 4.30 | 4.18 | 3.78 | 3.66 | 4.04 | 3.92 | 4.10 |258| MiMo‑Audio‑7B‑Instruct | 4.02 | 4.20 | 4.46 | 4.28 | 4.36 | 3.62 | 3.30 | 4.08 | 3.50 | 3.98 |259| Step‑Audio‑R1 | 4.22 | 4.02 | 4.20 | 3.96 | 3.84 | 4.02 | 3.24 | 4.10 | 3.54 | 3.90 |260| Qwen3‑Omni | 4.58 | 4.50 | 4.26 | 3.62 | 3.64 | 3.48 | 2.98 | 4.18 | 4.42 | 3.96 |261| Kimi‑Audio‑7B‑Instruct | 3.98 | 3.92 | 4.32 | 3.88 | 4.48 | 3.28 | 2.72 | 3.72 | 3.24 | 3.73 |262 263#### Song Describer Dataset (SDD)264 265| Model | Genre | Mood | Tempo | Instr. | Vocals | Melody/Harmony | Structure | Production | Scene | Avg |266|-----|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|267| **MOSS‑Music‑8B‑Instruct** | **4.84** | **4.76** | **4.68** | 4.24 | **4.52** | **4.56** | **4.92** | 4.42 | 4.24 | **4.58** |268| Gemini‑3.1‑Pro | 4.72 | 4.64 | 4.52 | **4.72** | 4.22 | 4.24 | 3.94 | **4.46** | **4.82** | 4.48 |269| **MOSS‑Music‑8B‑Thinking** | 4.66 | 4.58 | 4.50 | 4.36 | 4.36 | 4.44 | 4.84 | 4.26 | 4.02 | 4.45 |270| MusicFlamingo | 4.82 | 4.40 | 4.52 | 4.70 | 3.98 | 4.14 | 3.66 | 4.36 | 3.80 | 4.26 |271| Audio‑Flamingo‑Next | 4.40 | 4.62 | 4.14 | 4.36 | 4.22 | 3.84 | 3.74 | 4.10 | 4.00 | 4.16 |272| MiMo‑Audio‑7B‑Instruct | 4.08 | 4.26 | 4.52 | 4.34 | 4.42 | 3.70 | 3.38 | 4.16 | 3.58 | 4.05 |273| Step‑Audio‑R1 | 4.30 | 4.10 | 4.26 | 4.02 | 3.92 | 4.10 | 3.32 | 4.18 | 3.62 | 3.98 |274| Qwen3‑Omni | 4.62 | 4.54 | 4.30 | 3.68 | 3.70 | 3.56 | 3.06 | 4.24 | 4.50 | 4.02 |275| Kimi‑Audio‑7B‑Instruct | 4.04 | 3.98 | 4.38 | 3.96 | 4.54 | 3.36 | 2.80 | 3.80 | 3.32 | 3.80 |276 277### Lyrics ASR (WER / CER↓)278 279We further evaluate MOSS-Music on **singing-voice lyrics ASR** across three280representative benchmarks:281 282- `MUSDB18` — English pop songs **with backing tracks**, scored with **WER**;283- `MIR-1K` — **Chinese karaoke** clips with background music, scored with **CER**;284- `Opencpop` — **clean Mandarin studio singing**, scored with **CER**.285 286`Avg` is the unweighted mean of the three dataset-level error rates.287 288| Model | MUSDB18 WER | MIR-1K CER | Opencpop CER | Avg |289|-----|---:|---:|---:|---:|290| **MOSS‑Music‑8B‑Thinking** | 29.19% | **15.84%** | 2.60% | **15.88%** |291| **MOSS‑Music‑8B‑Instruct** | 32.99% | 23.96% | 4.62% | 20.52% |292| Gemini‑3.1‑Pro‑Preview | 26.25% | 36.37% | 6.00% | 22.87% |293| MusicFlamingo | **23.41%** | 38.98% | 18.73% | 27.04% |294| Qwen3‑Omni‑30B‑A3B‑Instruct | 62.67% | 20.48% | **2.26%** | 28.47% |295| MiMo‑Audio‑7B‑Instruct | 94.16% | 23.34% | 6.77% | 41.42% |296| Kimi‑Audio‑7B‑Instruct | 97.53% | 25.83% | 4.90% | 42.75% |297| Step‑Audio‑R1 | 81.67% | 48.03% | 4.15% | 44.62% |298| Audio‑Flamingo‑Next | 94.93% | 55.63% | 12.47% | 54.34% |299 300> **MOSS-Music-8B-Thinking** achieves the lowest average lyrics-ASR error301> (**15.88%**) across these three datasets, with particular gains on302> `MIR-1K` (Chinese karaoke with accompaniment) and `Opencpop` (clean Mandarin303> singing). MOSS-Music also inherits the strong timestamp-aware ASR ability304> from MOSS-Audio; detailed singing-timestamp ASR results will be added soon.305 306### Chord Transcription307 308MOSS-Music supports chord transcription, including timestamped chord309transcription that tracks chord progression over time. This can be useful for310harmonic analysis, accompaniment reference, music education, and related use311cases. Detailed benchmark results will be added soon.312 313## Quickstart314 315### Environment Setup316 317We recommend Python 3.12 with a clean Conda environment. The commands below318are enough for local inference.319 320#### Recommended setup321 322```bash323git clone https://github.com/OpenMOSS/MOSS-Music.git324cd MOSS-Music325 326conda create -n moss-music python=3.12 -y327conda activate moss-music328 329conda install -c conda-forge "ffmpeg=7" -y330pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime]"331```332 333#### Optional: FlashAttention 2334 335If your GPU supports FlashAttention 2, you can replace the last install336command with:337 338```bash339pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime,flash-attn]"340```341 342### SGLang Serving343 344> [!IMPORTANT]345> To achieve the best generation quality and fully leverage the model's capabilities, we346> **strongly recommend using SGLang Serving for inference**.347 348See the full SGLang guide in `moss_music_usage_guide.md`.349 350Download the model first:351 352```bash353hf download OpenMOSS-Team/MOSS-Music-8B-Instruct --local-dir ./weights/MOSS-Music-8B-Instruct354hf download OpenMOSS-Team/MOSS-Music-8B-Thinking --local-dir ./weights/MOSS-Music-8B-Thinking355```356 357The shortest setup is:358 359```bash360cd sglang361pip install -e "python[all]"362pip install nvidia-cudnn-cu12==9.16.0.29363cd ..364 365sglang serve \366 --model-path ./weights/MOSS-Music-8B-Instruct \367 --trust-remote-code368```369 370You can replace `./weights/MOSS-Music-8B-Instruct` with371`./weights/MOSS-Music-8B-Thinking` if needed.372 373If you use the default `torch==2.9.1+cu128` runtime, installing374`nvidia-cudnn-cu12==9.16.0.29` is recommended before starting `sglang serve`.375 376### Gradio App377 378Start the Gradio demo with:379 380```bash381python app.py382```383 384The server address and port can be overridden via the385`MOSS_MUSIC_SERVER_NAME` and `MOSS_MUSIC_SERVER_PORT` environment variables,386and the default model ID via `MOSS_MUSIC_MODEL_ID`.387 388## More Information389 390- **MOSI.AI**: [https://mosi.cn](https://mosi.cn)391- **OpenMOSS**: [https://www.open-moss.com](https://www.open-moss.com)392- **MOSS-Audio (backbone)**: [https://github.com/OpenMOSS/MOSS-Audio](https://github.com/OpenMOSS/MOSS-Audio)393- **MOSS-Music Data Pipeline**: [https://github.com/wx9songs/MOSS-Music-Data-Pipeline](https://github.com/wx9songs/MOSS-Music-Data-Pipeline)394 395## LICENSE396 397Models in MOSS-Music are licensed under the Apache License 2.0.398 399## Citation400 401```bibtex402@misc{mossmusic2026,403 title={MOSS-Music Technical Report},404 author={OpenMOSS Team},405 year={2026},406 howpublished={\url{https://github.com/OpenMOSS/MOSS-Music}},407 note={GitHub repository}408}409```410