FBK-MT/MCIF-ST
MCIF-ST: Context-aware Speech Recognition and Speech Translation from MCIF MCIF-ST provides both long-form and short-form ready-to-use Automatic Speech Recogniton (ASR) and Speech Translation (ST) data derived from MCIF (Multimodal Crosslingual Instruction Following), a multilingual benchmark based on scientific talks. While the original MCIF release packages its content as instruction-following rows (multimodal context + prompt + expected answer, for… See the full description on the dataset page: https://huggingface.co/datasets/FBK-MT/MCIF-ST.
<p align="center"> <img src="https://huggingface.co/datasets/FBK-MT/MCIF/resolve/main/mcif_logo.png" width="600"> </p>
MCIF-ST: Context-aware Speech Recognition and Speech Translation from MCIF
MCIF-ST provides both long-form and short-form ready-to-use Automatic Speech Recogniton (ASR) and Speech Translation (ST) data derived from MCIF (Multimodal Crosslingual Instruction Following), a multilingual benchmark based on scientific talks. While the original MCIF release packages its content as instruction-following rows (multimodal context + prompt + expected answer, for QA/Summarization/ASR/Translation tasks), MCIF-ST re-exposes the same underlying talk- and sentence-level alignments as plain (audio, source transcript, reference translations) rows -- no instruction wrapper -- for directly evaluating ASR/ST systems, the way you'd use LibriSpeech or CoVoST2.
Two configs are provided over the same 919 sentence-level segments across 21 talks (about 2 hours), differing only in how audio (and, for ST_long, which also contain the paper pdf, useful for context-aware evaluation) is packaged:
- `ST_short`: audio is physically cut into one clip per segment.
- `ST_long`: audio is kept as the full, unsegmented talk recording, with
offset/durationlocating each segment, plus the talk's source pdf.
Dataset Structure (ST_short)
Dataset Structure (ST_long)
Usage
from datasets import load_dataset
# ST_short: pre-cut clips, load and use directly
ds_short = load_dataset("FBK-MT/MCIF-ST", "ST_short", split="test")
row = ds_short[0]
print(row["source_text_en"], "->", row["target_text_de"])
# ST_long: full audio, slice using offset/duration if you want to do short-form inference
# otherwise, load directly the files contained in MCIF_DATA/LONG_AUDIO (read below)
import soundfile as sf
ds_long = load_dataset("FBK-MT/MCIF-ST", "ST_long", split="test")
row = ds_long[0]
audio, sr = sf.read(row["audio"])
start, end = int(row["offset"] * sr), int((row["offset"] + row["duration"]) * sr)
segment_audio = audio[start:end]For an easy evaluation with resegmentation tools (e.g., mWERsegmenter), audio_segments.yaml file is provided containing the segments alignment of the long-form audio contained in MCIF_DATA/LONG_AUDIO (with also the corresponding pdf files for each talk in MCIF_DATA/PDF), together with {en,de,it,zh}.txt reference files, with one segment per line.
License
CC-BY-4.0, matching the source MCIF release.
Citation
If you use this data, please cite the original MCIF paper:
@inproceedings{papi2026mcif,
title={{MCIF}: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks},
author={Sara Papi and Maike Z{\"u}fle and Marco Gaido and Beatrice Savoldi and Danni Liu and Ioannis Douros and Luisa Bentivogli and Jan Niehues},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=PtPYZYfa0h}
}