jupyterjazz/XModBench-MTEB

XModBench-Lite for MTEB

This repository is a deterministic MTEB normalization of the official `RyanWW/XModBench` XModBench-Lite release at revision a679188cf062b9810d2e09c2edabc0b1aef9f244. The source contains 6,000 four-choice questions balanced across six canonical modality configurations and five capability families.

This MTEB adaptation retains 5,981 questions. It excludes 19 questions that reference five unusable MP4 files in the pinned official archive. Four are truncated: a7cRojOdljw.mp4, hPuylJBmk_8.mp4, sFnX5gB99r8.mp4, and uby2dcP6cmw.mp4. The files lack a final MP4 index and their video payloads end before the corresponding audio. The fifth, rivera0923_00_9_2.95_10.00.mp4, cannot be sought to its first presentation timestamp by TorchCodec 0.14. The exclusions configuration records every omitted source row, media path, usage, and reason. Original source indices are preserved in retained IDs.

Each question is represented as a reranking problem with four candidates, one relevant document, and a top_ranked list that restricts evaluation to the original answer choices. Accuracy is therefore equivalent to the source multiple-choice metric.

MTEB assigns modalities at task level, while XModBench uses Vision to mean the union of Image and Video. This normalization consequently exposes ten concrete directions: at2t, at2i, at2v, t2a, t2i, t2v, it2a, vt2a, it2t, and vt2t. Each direction has queries, corpus, qrels, and top_ranked configurations. The metadata configuration preserves source indices, canonical configurations, families, subtasks, categories, modalities, answers, and original questions; exclusions documents the 19 omitted rows.

Query-side media are accompanied by the semantic question text. For text conditions, the query is formatted as Context: {condition} followed by the source question, matching the authors' lmms-eval integration. The conversion removes only the source's exact trailing A/B/C/D answer-format boilerplate.

Reproducibility

Generated by scripts/data/xmodbench/create_data.py in MTEB from:

• —dataset: RyanWW/XModBench@a679188cf062b9810d2e09c2edabc0b1aef9f244
• —source code: `XingruiWang/XModBench@687a9230f3621f96eeca21f8c873418c57411022`
• —Lite builder: `XingruiWang/lmms-eval@313bf8034184ad245fa10939080692e377dac0e3`

License and citation

The source benchmark is released under the MIT License. Its authors note that redistributed media remain subject to the licenses of their underlying source datasets. Please review the source dataset card before reuse.

@inproceedings{wang2026xmodbench,
  title     = {XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models},
  author    = {Wang, Xingrui and Liu, Jiang and Huang, Chao and Yu, Xiaodong and Wang, Ze and Sun, Ximeng and Wu, Jialian and Yuille, Alan and Barsoum, Emad and Liu, Zicheng},
  booktitle = {International Conference on Learning Representations (ICLR)},
  year      = {2026},
  url       = {https://arxiv.org/abs/2510.15148}
}