BUT-FIT/DiCoW_v3_MLC
DiCoW\v3\MLC — BUT-FIT Model for MLC-SLM Challenge
This repository contains the DiCoW\_v3\_MLC model developed by BUT Speech@FIT for the MLC-SLM Challenge. Diarization-Conditioned Whisper (DiCoW) is a novel approach to target-speaker ASR that leverages speaker diarization outputs as conditioning information.
This model is available under the terms of CC BY 4.0. It incorporates an MIT-licensed base model and CC BY 4.0 licensed training data.
The model is described in detail in the following papers:
- 📰 Journal paper (main DiCoW paper): DiCoW: Diarization-Conditioned Whisper for Target Speaker Automatic Speech Recognition
- 📰 ICASSP paper (initial DiCoW experiments): Target Speaker ASR with Whisper
- 📰 MLC-SLM Challenge submission paper: BUT System for the MLC-SLM Challenge
Model Summary
The model is based on Whisper large-v3-turbo, initially trained on:
- NOTSOFAR-1
- AMI Meeting Corpus
- Libri2Mix dataset
It is then fine-tuned on the MLC-SLM dataset as part of the MLC-SLM Challenge.
Model Details
- Developed by: BUT Speech\@FIT, Brno University of Technology
- Model type: Whisper large-v3-turbo + DiCoW composition
- Language(s): Multilingual (primarily English, but supports multiple languages)
- License: apache-2.0
- Fine-tuned from: openai/whisper-large-v3-turbo
- Challenge: MLC-SLM (Multilingual Conversational Speech Language Model)
Model Sources
- Training Code: TS-ASR-Whisper GitHub
- Inference Code & DiCoW framework: DiCoW GitHub
Getting Started
from transformers import AutoModelForSpeechSeq2Seq
MODEL_NAME = "BUT-FIT/DiCoW_v3_MLC"
dicow = AutoModelForSpeechSeq2Seq.from_pretrained(MODEL_NAME, trust_remote_code=True)For detailed inference and full pipelines, refer to: 👉 DiCoW GitHub inference repo
tcpWER/CER (%) on the MLC-SLM development set
Results marked with an asterisk () are reported using tcpCER, following the official evaluation protocol.*
Notes:
- GT = Ground-Truth Segmentation
- Real diar = Real Diarization
- Baseline uses Whisper large-v3 with chunked inference + finetunned Pyannote diarization.
- DiCoW uses fine-tuned DiariZen diarization.
Citation
If you use this model, please cite:
@article{POLOK2026101841,
title = {DiCoW: Diarization-conditioned Whisper for target speaker automatic speech recognition},
journal = {Computer Speech & Language},
volume = {95},
pages = {101841},
year = {2026},
issn = {0885-2308},
doi = {https://doi.org/10.1016/j.csl.2025.101841},
url = {https://www.sciencedirect.com/science/article/pii/S088523082500066X},
author = {Alexander Polok and Dominik Klement and Martin Kocour and Jiangyu Han and Federico Landini and Bolaji Yusuf and Matthew Wiesner and Sanjeev Khudanpur and Jan Černocký and Lukáš Burget},
keywords = {Diarization-conditioned Whisper, Target-speaker ASR, Speaker diarization, Long-form ASR, Whisper adaptation},
}
@INPROCEEDINGS{10887683,
author={Polok, Alexander and Klement, Dominik and Wiesner, Matthew and Khudanpur, Sanjeev and Černocký, Jan and Burget, Lukáš},
booktitle={ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
title={Target Speaker ASR with Whisper},
year={2025},
volume={},
number={},
pages={1-5},
keywords={Transforms;Signal processing;Transformers;Acoustics;Speech processing;target-speaker ASR;diarization conditioning;multi-speaker ASR;Whisper},
doi={10.1109/ICASSP49660.2025.10887683}
}
@misc{polok2025mlcslmchallenge,
title={BUT System for the MLC-SLM Challenge},
author={Alexander Polok and Jiangyu Han and Dominik Klement and Samuele Cornell and Jan Černocký and Lukáš Burget},
year={2025},
eprint={2506.13414},
archivePrefix={arXiv},
primaryClass={eess.AS},
url={https://arxiv.org/abs/2506.13414},
}Contact
For questions or collaborations, feel free to email: ipoloka@fit.vut.cz BUT Speech@FIT, Brno University of Technology GitHub: BUTSpeechFIT