CoolFace
Datasetpublic

vaishnavikedar4/MCIF

Dataset Description, Collection, and Source MCIF (Multimodal Crosslingual Instruction Following) is a multilingual human-annotated benchmark based on scientific talks that is designed to evaluate instruction-following in crosslingual, multimodal settings over both short- and long-form inputs. MCIF spans three core modalities -- speech, vision, and text -- and four diverse languages (English, German, Italian, and Chinese), enabling a comprehensive evaluation of MLLMs'… See the full description on the dataset page: https://huggingface.co/datasets/vaishnavikedar4/MCIF.

sourceHugging Facecc-by-4.0updated 9mo agoView on Hugging Face
0likes41downloads
Dataset Card

<p align="center"> <img src="./mcif_logo.png" width="600"> </p>

Dataset Description, Collection, and Source

MCIF (Multimodal Crosslingual Instruction Following) is a multilingual human-annotated benchmark based on scientific talks that is designed to evaluate instruction-following in crosslingual, multimodal settings over both short- and long-form inputs. MCIF spans three core modalities -- speech, vision, and text -- and four diverse languages (English, German, Italian, and Chinese), enabling a comprehensive evaluation of MLLMs' abilities to interpret instructions across languages and combine them with multimodal contextual information.

License

  • —CC-BY-4.0

Dataset Sources

Dataset Structure

Data Config

This dataset contains 4 splits organized by three dimensions following the split naming convention {track}_{prompt_type}.

Track - Input duration:

  • —long: Full-length, unsegmented inputs
  • —short: Pre-segmented inputs

Prompt Type - Prompt variation:

  • —fixed: Standardized prompts across all examples
  • —mixed: Includes prompt variations

Please note that all splits share the same set of original input audio and video files. The splits are meant to facilitate testing various use cases.

Dataset Fields

**Field****Type****Description**
idstringUnique identifier for the sample, it starts with QA (question answering), SUM (summarization), ASR (transcription), or TRANS (translation).
audiostrIn the long track: path to full talk-level audio. In the short track: path to pre-segmented audio.
videostrIn the long track: path to full talk-level video. In the short track: path to pre-segmented video.
textstringTranscript of input. Only present in the long track.
prompt_{en, de, it, zh}stringInstruction in English, German, Italian, or Chinese.
metadatastringMeta data for question answering samples, in the format {qatype={`A` (audio), `V` (visual), `AV` (audio-visual), `NA` (not answerable)} qaorigin={Transcript, Abstract, General}}

The audio/video paths are relative within this repo.

You can download the data by cloning this repo:

git clone https://huggingface.co/datasets/FBK-MT/MCIF

References

The references are available in MCIF.{short,long}.{en,de,it,zh}.ref.xml.gz (navigate to "Files and versions" tab or clone this repo).

IWSLT 2025 Version

Part of MCIF was used in the IWSLT 2025 instruction-following track.

This test data is available under branch IWSLT2025. You can access it by 

dataset = load_dataset("FBK-MT/MCIF", "{en,de,it,zh}_{long,short}", revision="IWSLT2025")

Evaluation

Please use the official evaluation scripts from the MCIF GitHub Repo. The references are also available there.

Changelog

Version 1.2

  • —Fixed summarization references

Version 1.1

  • —Fixed German summarization prompt
  • —Renamed files not to include version name in the filename

Citation

@misc{papi2025mcifmultimodalcrosslingualinstructionfollowing,
      title={MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks}, 
      author={Sara Papi and Maike Züfle and Marco Gaido and Beatrice Savoldi and Danni Liu and Ioannis Douros and Luisa Bentivogli and Jan Niehues},
      year={2025},
      eprint={2507.19634},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2507.19634}, 
}

Dataset Card Contact

@spapi and @danniliu