ASLP-lab/CN-MultiDialect-ASR

<p align="center"> <img src="https://github.com/ASLP-lab/CN-MultiDialect-ASR/raw/main/assets/logo.jpeg" width="520" alt="CN-MultiDialect-ASR logo"> </p>

On-Policy Self-Distillation for Multi-Dialect ASR: Mastering Dialects, Retaining Mandarin

<div align="center"> <p><strong>Shuiyuan Wang¹ · Bingshen Mu¹ · Pengshen Zhang² · Chengyou Wang¹ · Yujie Liao¹ · Chengdong Liang² · Binbin Zhang² · Qiangze Feng³ · Lei Xie¹</strong></p> <p>¹ Audio, Speech and Language Processing Group (ASLP@NPU), School of Computer Science, Northwestern Polytechnical University, Xi'an, China<br> ² WeNet Community<br> ³ NEXDATA TECHNOLOGY INC.</p> </div>

<div align="center">

  

</div>

This repository hosts the released CN-MultiDialect-ASR checkpoint, adapted from Qwen3-ASR-1.7B with a three-stage pipeline: continual pre-training (CPT), dialect supervised fine-tuning (SFT), and On-Policy Self-Distillation (OPSD). The goal is to improve Chinese dialect recognition without raising Mandarin CER.

• —Paper: arXiv:2608.11898
• —Code, demo, and training scripts: ASLP-lab/CN-MultiDialect-ASR

<div align="center"> <img src="https://github.com/ASLP-lab/CN-MultiDialect-ASR/raw/main/assets/opsd.png" alt="OPSD framework" width="90%"> <p><em>Overview of the staged adaptation pipeline. Top: base model, CPT, SFT, and OPSD. Bottom: OPSD with student on-policy prefixes, a frozen teacher conditioned on the reference transcript as privileged context, soft targets q_t, and token-level KL.</em></p> </div>

Demo

Video demo with live waveforms and model transcriptions for Mandarin, English, four core dialects, and 15 ChinaVoices dialects.

<video src="https://github.com/user-attachments/assets/29439247-bc62-45e1-8119-e0c45416c957" controls preload="metadata" playsinline width="100%" aria-label="CN-MultiDialect-ASR video demo"></video>

Key Features

• —Mandarin–dialect balanced adaptation: improves Chinese dialect ASR while retaining Mandarin recognition.
• —Three-stage pipeline: CPT strengthens the Chinese ASR foundation, dialect SFT specializes for dialects, and OPSD refines the final checkpoint.
• —On-Policy Self-Distillation: trains on student-decoded prefixes with soft teacher targets, reducing the train–test mismatch of teacher-forced ASR training.
• —Drop-in inference: compatible with the official `qwen-asr` package.

Quickstart

Inference is compatible with Qwen3-ASR. We recommend installing the official qwen-asr package in a clean environment.

Environment Setup

conda create -n qwen3-asr python=3.12 -y
conda activate qwen3-asr
pip install -U qwen-asr

For faster inference with the vLLM backend:

pip install -U qwen-asr[vllm]

Model Download

You can load the model directly from Hugging Face, or download it locally first:

# Hugging Face
pip install -U "huggingface_hub[cli]"
hf download ASLP-lab/CN-MultiDialect-ASR --local-dir ./CN-MultiDialect-ASR

# ModelScope (recommended for users in Mainland China)
pip install -U modelscope
modelscope download --model ASLP-lab/CN-MultiDialect-ASR --local_dir ./CN-MultiDialect-ASR

Python Inference

Load the model with Qwen3ASRModel.from_pretrained and call transcribe:

import torch
from qwen_asr import Qwen3ASRModel

model = Qwen3ASRModel.from_pretrained(
    "ASLP-lab/CN-MultiDialect-ASR",  # or "./CN-MultiDialect-ASR" for a local path
    dtype=torch.bfloat16,
    device_map="cuda:0",
    # attn_implementation="flash_attention_2",
    max_inference_batch_size=32,
    max_new_tokens=256,
)

results = model.transcribe(
    audio="path/to/audio.wav",
    language="Chinese",  # or None for automatic language detection
)

print(results[0].language)
print(results[0].text)

Batch inference is also supported:

results = model.transcribe(
    audio=[
        "path/to/mandarin.wav",
        "path/to/dialect.wav",
    ],
    language=["Chinese", "Chinese"],
)

for r in results:
    print(r.language, r.text)

For vLLM backend, streaming inference, and forced alignment, see the Qwen3-ASR repository.

Method Overview

At inference time, only the student pathway is used.

Performances

Dialect Overview

<div align="center"> <img src="https://github.com/ASLP-lab/CN-MultiDialect-ASR/raw/main/assets/radar1cer_panels.png" alt="Side-by-side radar of 1-CER on public and internal dialect sets" width="92%"> <p><em>Higher is better. Left: 5 public dialect sets; right: 18 internal dialects. Both panels use the same radial scale (0.2–1.0). The figure compares the Qwen3-ASR baseline with the released <strong>CN-MultiDialect-ASR</strong> (OPSD) checkpoint.</em></p> </div>

Public Dialect CER (%)

<table> <thead> <tr> <th align="center">Evaluation set</th> <th align="center">Dialect</th> <th align="center">Qwen3-ASR</th> <th align="center">CN-MultiDialect-ASR</th> </tr> </thead> <tbody> <tr> <td align="center">WenetSpeech-Yue Long</td> <td align="center">Cantonese</td> <td align="center">9.99</td> <td align="center"><b>8.80</b></td> </tr> <tr> <td align="center">WenetSpeech-Yue Short</td> <td align="center">Cantonese</td> <td align="center">6.93</td> <td align="center"><b>5.31</b></td> </tr> <tr> <td align="center">WenetSpeech-Chuan Easy</td> <td align="center">Sichuan</td> <td align="center">12.38</td> <td align="center"><b>11.86</b></td> </tr> <tr> <td align="center">WenetSpeech-Chuan Hard</td> <td align="center">Sichuan</td> <td align="center">21.79</td> <td align="center"><b>21.74</b></td> </tr> <tr> <td align="center">WenetSpeech-Wu</td> <td align="center">Wu</td> <td align="center">25.74</td> <td align="center"><b>16.26</b></td> </tr> <tr> <td align="center"><b>Dialect Avg.</b></td> <td align="center"></td> <td align="center">15.37</td> <td align="center"><b>12.79</b></td> </tr> </tbody> </table>

Internal Dialect CER (%)

<table> <thead> <tr> <th align="center">Dialect</th> <th align="center">Qwen3-ASR</th> <th align="center">CN-MultiDialect-ASR</th> </tr> </thead> <tbody> <tr><td align="center">Anhui</td><td align="center">18.95</td><td align="center"><b>13.08</b></td></tr> <tr><td align="center">Cantonese</td><td align="center">10.06</td><td align="center"><b>7.74</b></td></tr> <tr><td align="center">Changsha</td><td align="center">14.79</td><td align="center"><b>10.23</b></td></tr> <tr><td align="center">Chaoshan</td><td align="center">45.59</td><td align="center"><b>25.21</b></td></tr> <tr><td align="center">Dongbei</td><td align="center">6.45</td><td align="center"><b>5.80</b></td></tr> <tr><td align="center">Henan</td><td align="center">8.46</td><td align="center"><b>5.99</b></td></tr> <tr><td align="center">Kejia</td><td align="center">60.47</td><td align="center"><b>28.60</b></td></tr> <tr><td align="center">Minnan</td><td align="center">30.03</td><td align="center"><b>18.59</b></td></tr> <tr><td align="center">Nanchang</td><td align="center">33.41</td><td align="center"><b>15.58</b></td></tr> <tr><td align="center">Nanjing</td><td align="center">13.37</td><td align="center"><b>9.33</b></td></tr> <tr><td align="center">Shanxi</td><td align="center">28.53</td><td align="center"><b>18.69</b></td></tr> <tr><td align="center">Shaanxi</td><td align="center">9.68</td><td align="center"><b>6.28</b></td></tr> <tr><td align="center">Shandong</td><td align="center">8.78</td><td align="center"><b>7.64</b></td></tr> <tr><td align="center">Shanghai</td><td align="center">15.78</td><td align="center">12.07</td></tr> <tr><td align="center">Sichuan</td><td align="center">5.99</td><td align="center">5.38</td></tr> <tr><td align="center">Suzhou</td><td align="center">50.35</td><td align="center"><b>20.73</b></td></tr> <tr><td align="center">Wuhan</td><td align="center">11.30</td><td align="center"><b>7.59</b></td></tr> <tr><td align="center">Xuzhou</td><td align="center">6.12</td><td align="center"><b>5.04</b></td></tr> <tr><td align="center"><b>Internal Avg.</b></td><td align="center">21.01</td><td align="center"><b>12.42</b></td></tr> </tbody> </table>

Mandarin CER (%)

<table> <thead> <tr> <th align="center">Evaluation set</th> <th align="center">Qwen3-ASR</th> <th align="center">CN-MultiDialect-ASR</th> </tr> </thead> <tbody> <tr><td align="center">AISHELL-1</td><td align="center">1.57</td><td align="center"><b>1.38</b></td></tr> <tr><td align="center">AISHELL-2</td><td align="center">2.79</td><td align="center"><b>2.52</b></td></tr> <tr><td align="center">KeSpeech</td><td align="center">5.11</td><td align="center"><b>4.56</b></td></tr> <tr><td align="center">SpeechIO-1</td><td align="center"><b>0.75</b></td><td align="center">0.86</td></tr> <tr><td align="center">SpeechIO-2</td><td align="center">3.83</td><td align="center"><b>3.39</b></td></tr> <tr><td align="center">SpeechIO-3</td><td align="center">1.39</td><td align="center"><b>1.27</b></td></tr> <tr><td align="center">TestMeeting</td><td align="center"><b>6.74</b></td><td align="center">6.85</td></tr> <tr><td align="center">TestNet</td><td align="center">5.46</td><td align="center"><b>5.30</b></td></tr> <tr><td align="center"><b>Mandarin Avg.</b></td><td align="center">3.46</td><td align="center"><b>3.27</b></td></tr> </tbody> </table>

Citation

If you use this model, please cite:

@misc{wang2026onpolicyselfdistillationmultidialectasr,
  title={On-Policy Self-Distillation for Multi-Dialect ASR: Mastering Dialects, Retaining Mandarin},
  author={Shuiyuan Wang and Bingshen Mu and Pengshen Zhang and Chengyou Wang and Yujie Liao and Chengdong Liang and Binbin Zhang and Qiangze Feng and Lei Xie},
  year={2026},
  eprint={2608.11898},
  archivePrefix={arXiv},
  primaryClass={eess.AS},
  url={https://arxiv.org/abs/2608.11898}
}

License

The released model is licensed under Apache 2.0.

Contact

For questions or collaborations, please contact wangshuiyuan@mail.nwpu.edu.cn.

You are also welcome to join our WeChat group for technical discussions and updates.

<p align="center"> <img src="https://github.com/ASLP-lab/CN-MultiDialect-ASR/raw/main/assets/wechat.jpg" width="300" alt="WeChat group QR code"> <br> <em>Scan to join our WeChat discussion group</em> </p>