RunsenXu/MMSI-Bench

1---2language:3- en4license: cc-by-4.05size_categories:6- 1K<n<10K7task_categories:8- question-answering9- visual-question-answering10- multiple-choice11pretty_name: MMSI-Bench12dataset_info:13  features:14  - name: id15    dtype: int6416  - name: images17    sequence: image18  - name: question_type19    dtype: string20  - name: question21    dtype: string22  - name: answer23    dtype: string24  - name: thought25    dtype: string26  - name: mean_normed_duration_seconds27    dtype: float6428  - name: difficulty29    dtype:30      class_label:31        names:32          '0': easy33          '1': medium34          '2': hard35  splits:36    - name: test37      num_examples: 100038 39configs:40  - config_name: default41    data_files:42      - split: test43        path: MMSI_Bench.parquet44---45 46# MMSI-Bench47This repo contains evaluation code for the paper "MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence" 48 49[**🌐 Homepage**](https://runsenxu.com/projects/MMSI_Bench/) | [**🤗 Dataset**](https://huggingface.co/datasets/RunsenXu/MMSI-Bench) | [**📑 Paper**](https://arxiv.org/pdf/2505.23764) | [**💻 Code**](https://github.com/OpenRobotLab/MMSI-Bench) | [**📖 arXiv**](https://arxiv.org/abs/2505.23764)50 51 52 53## 🔔News54  **🔥[2025-10-23]: We added the normalized human response time for each MMSI-Bench sample and its difficulty level to our dataset on Hugging Face.**55 56  **🔥[2025-06-18]: MMSI-Bench has been supported in the [LMMs-Eval](https://github.com/EvolvingLMMs-Lab/lmms-eval) repository.**57 58  **✨[2025-06-11]: MMSI-Bench was used for evaluation in the experiments of [VILASR](https://arxiv.org/abs/2506.09965).**59 60  **🔥[2025-06-9]: MMSI-Bench has been supported in the [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) repository.**61 62  **🔥[2025-05-30]: We released the ArXiv paper.**63 64## Load Dataset65```66from datasets import load_dataset67 68mmsi_bench = load_dataset("RunsenXu/MMSI-Bench")69print(mmsi_bench)70```71 72## After downloading the parquet file, read each record, decode images from binary, and save them as JPG files.73```74import pandas as pd75import os76 77df = pd.read_parquet('MMSI_Bench.parquet')78 79output_dir = './images'80os.makedirs(output_dir, exist_ok=True)81 82for idx, row in df.iterrows():83    id_val = row['id']84    images = row['images']  85    question_type = row['question_type']86    question = row['question']87    answer = row['answer']88    thought = row['thought']89 90    image_paths = []91    if images is not None:92        for n, img_data in enumerate(images):93            image_path = f"{output_dir}/{id_val}_{n}.jpg"94            with open(image_path, "wb") as f:95                f.write(img_data)96            image_paths.append(image_path)97    else:98        image_paths = []99 100    print(f"id: {id_val}")101    print(f"images: {image_paths}")102    print(f"question_type: {question_type}")103    print(f"question: {question}")104    print(f"answer: {answer}")105    print(f"thought: {thought}")106    print("-" * 50)107```108 109## Evaluation110Please refer to the [evaluation guidelines](https://github.com/open-compass/VLMEvalKit/blob/main/docs/en/Quickstart.md) of [VLMEvalKit](https://github.com/open-compass/VLMEvalKit)111 112<!-- <img src="assets/radar_v1.png" width="400" /> -->113 114## 🏆 MMSI-Bench Leaderboard115 116| Model                        | Avg. (%) | Type         |117|------------------------------|:--------:|:-------------|118| 🥇 **Human Level**           | 97.2     | Baseline     |119| 🥈 o3                        | 41.0     | Proprietary  |120| 🥉 GPT-4.5                   | 40.3     | Proprietary  |121| Gemini-2.5-Pro--Thinking     | 37.0     | Proprietary  |122| Gemini-2.5-Pro               | 36.9     | Proprietary  |123| Doubao-1.5-pro               | 33.0     | Proprietary  |124| GPT-4.1                      | 30.9     | Proprietary  |125| Qwen2.5-VL-72B               | 30.7     | Open-source  |126| NVILA-15B                    | 30.5     | Open-source  |127| GPT-4o                       | 30.3     | Proprietary  |128| Claude-3.7-Sonnet--Thinking  | 30.2     | Proprietary  |129| Seed1.5-VL                   | 29.7     | Proprietary  |130| InternVL2.5-2B               | 29.0     | Open-source  |131| InternVL2.5-8B               | 28.7     | Open-source  |132| DeepSeek-VL2-Small           | 28.6     | Open-source  |133| InternVL3-78B                | 28.5     | Open-source  |134| InternVL2.5-78B              | 28.5     | Open-source  |135| LLaVA-OneVision-72B          | 28.4     | Open-source  |136| NVILA-8B                     | 28.1     | Open-source  |137| InternVL2.5-26B              | 28.0     | Open-source  |138| DeepSeek-VL2                 | 27.1     | Open-source  |139| InternVL3-1B                 | 27.0     | Open-source  |140| InternVL3-9B                 | 26.7     | Open-source  |141| Qwen2.5-VL-3B                | 26.5     | Open-source  |142| InternVL2.5-1B               | 26.1     | Open-source  |143| InternVL2.5-4B               | 26.3     | Open-source  |144| Qwen2.5-VL-7B                | 25.9     | Open-source  |145| InternVL3-8B                 | 25.7     | Open-source  |146| Llama-3.2-11B-Vision         | 25.4     | Open-source  |147| InternVL3-2B                 | 25.3     | Open-source  |148| 🃏 **Random Guessing**        | 25.0     | Baseline     |149| LLaVA-OneVision-7B           | 24.5     | Open-source  |150| DeepSeek-VL2-Tiny            | 24.0     | Open-source  |151| Blind GPT-4o                 | 22.7     | Baseline     |152 153## Acknowledgment154MMSI-Bench makes use of data from existing image datasets: [ScanNet](http://www.scan-net.org/), [nuScenes](https://www.nuscenes.org/), [Matterport3D](https://niessner.github.io/Matterport/), [Ego4D](https://ego4d-data.org/), [AgiBot-World](https://agibot-world.cn/), [DTU](https://roboimagedata.compute.dtu.dk/?page_id=36), [DAVIS-2017](https://davischallenge.org/) ,and [Waymo](https://waymo.com/open/). We thank these teams for their open-source contributions.155 156## Contact157- Sihan Yang: sihany077@gmail.com158- Runsen Xu:  runsxu@gmail.com159 160## Citation161```bibtex162@article{yang2025mmsi,163  title={MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence},164  author={Yang, Sihan and Xu, Runsen and Xie, Yiman and Yang, Sizhe and Li, Mo and Lin, Jingli and Zhu, Chenming and Chen, Xiaochen and Duan, Haodong and Yue, Xiangyu and Lin, Dahua and Wang, Tai and Pang, Jiangmiao},165  journal={arXiv preprint arXiv:2505.23764},166  year={2025}167}168```