RunsenXu/MMSI-Bench
MMSI-Bench This repo contains evaluation code for the paper "MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence" π Homepage | π€ Dataset | π Paper | π» Code | π arXiv πNews π₯[2025-10-23]: We added the normalized human response time for each MMSI-Bench sample and its difficulty level to our dataset on Hugging Face. π₯[2025-06-18]: MMSI-Bench has been supported in the LMMs-Eval repository. β¨[2025-06-11]: MMSI-Bench was used for evaluation inβ¦ See the full description on the dataset page: https://huggingface.co/datasets/RunsenXu/MMSI-Bench.
MMSI-Bench
This repo contains evaluation code for the paper "MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence"
**π Homepage** | **π€ Dataset** | **π Paper** | **π» Code** | **π arXiv**
πNews
π₯[2025-10-23]: We added the normalized human response time for each MMSI-Bench sample and its difficulty level to our dataset on Hugging Face.
π₯[2025-06-18]: MMSI-Bench has been supported in the [LMMs-Eval](https://github.com/EvolvingLMMs-Lab/lmms-eval) repository.
β¨[2025-06-11]: MMSI-Bench was used for evaluation in the experiments of [VILASR](https://arxiv.org/abs/2506.09965).
π₯[2025-06-9]: MMSI-Bench has been supported in the [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) repository.
π₯[2025-05-30]: We released the ArXiv paper.
Load Dataset
from datasets import load_dataset
mmsi_bench = load_dataset("RunsenXu/MMSI-Bench")
print(mmsi_bench)After downloading the parquet file, read each record, decode images from binary, and save them as JPG files.
import pandas as pd
import os
df = pd.read_parquet('MMSI_Bench.parquet')
output_dir = './images'
os.makedirs(output_dir, exist_ok=True)
for idx, row in df.iterrows():
id_val = row['id']
images = row['images']
question_type = row['question_type']
question = row['question']
answer = row['answer']
thought = row['thought']
image_paths = []
if images is not None:
for n, img_data in enumerate(images):
image_path = f"{output_dir}/{id_val}_{n}.jpg"
with open(image_path, "wb") as f:
f.write(img_data)
image_paths.append(image_path)
else:
image_paths = []
print(f"id: {id_val}")
print(f"images: {image_paths}")
print(f"question_type: {question_type}")
print(f"question: {question}")
print(f"answer: {answer}")
print(f"thought: {thought}")
print("-" * 50)Evaluation
Please refer to the evaluation guidelines of VLMEvalKit
<!-- <img src="assets/radar_v1.png" width="400" /> -->
π MMSI-Bench Leaderboard
Acknowledgment
MMSI-Bench makes use of data from existing image datasets: ScanNet, nuScenes, Matterport3D, Ego4D, AgiBot-World, DTU, DAVIS-2017 ,and Waymo. We thank these teams for their open-source contributions.
Contact
- Sihan Yang: sihany077@gmail.com
- Runsen Xu: runsxu@gmail.com
Citation
@article{yang2025mmsi,
title={MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence},
author={Yang, Sihan and Xu, Runsen and Xie, Yiman and Yang, Sizhe and Li, Mo and Lin, Jingli and Zhu, Chenming and Chen, Xiaochen and Duan, Haodong and Yue, Xiangyu and Lin, Dahua and Wang, Tai and Pang, Jiangmiao},
journal={arXiv preprint arXiv:2505.23764},
year={2025}
}