CoolFace
Datasetpublic

RunsenXu/MMSI-Bench

MMSI-Bench This repo contains evaluation code for the paper "MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence" 🌐 Homepage | πŸ€— Dataset | πŸ“‘ Paper | πŸ’» Code | πŸ“– arXiv πŸ””News πŸ”₯[2025-10-23]: We added the normalized human response time for each MMSI-Bench sample and its difficulty level to our dataset on Hugging Face. πŸ”₯[2025-06-18]: MMSI-Bench has been supported in the LMMs-Eval repository. ✨[2025-06-11]: MMSI-Bench was used for evaluation in… See the full description on the dataset page: https://huggingface.co/datasets/RunsenXu/MMSI-Bench.

sourceHugging Facecc-by-4.0updated 11mo agoView on Hugging Face
17likes1.5kdownloads
Dataset Card

MMSI-Bench

This repo contains evaluation code for the paper "MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence"

**🌐 Homepage** | **πŸ€— Dataset** | **πŸ“‘ Paper** | **πŸ’» Code** | **πŸ“– arXiv**

πŸ””News

πŸ”₯[2025-10-23]: We added the normalized human response time for each MMSI-Bench sample and its difficulty level to our dataset on Hugging Face.

πŸ”₯[2025-06-18]: MMSI-Bench has been supported in the [LMMs-Eval](https://github.com/EvolvingLMMs-Lab/lmms-eval) repository.

✨[2025-06-11]: MMSI-Bench was used for evaluation in the experiments of [VILASR](https://arxiv.org/abs/2506.09965).

πŸ”₯[2025-06-9]: MMSI-Bench has been supported in the [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) repository.

πŸ”₯[2025-05-30]: We released the ArXiv paper.

Load Dataset

from datasets import load_dataset

mmsi_bench = load_dataset("RunsenXu/MMSI-Bench")
print(mmsi_bench)

After downloading the parquet file, read each record, decode images from binary, and save them as JPG files.

import pandas as pd
import os

df = pd.read_parquet('MMSI_Bench.parquet')

output_dir = './images'
os.makedirs(output_dir, exist_ok=True)

for idx, row in df.iterrows():
    id_val = row['id']
    images = row['images']  
    question_type = row['question_type']
    question = row['question']
    answer = row['answer']
    thought = row['thought']

    image_paths = []
    if images is not None:
        for n, img_data in enumerate(images):
            image_path = f"{output_dir}/{id_val}_{n}.jpg"
            with open(image_path, "wb") as f:
                f.write(img_data)
            image_paths.append(image_path)
    else:
        image_paths = []

    print(f"id: {id_val}")
    print(f"images: {image_paths}")
    print(f"question_type: {question_type}")
    print(f"question: {question}")
    print(f"answer: {answer}")
    print(f"thought: {thought}")
    print("-" * 50)

Evaluation

Please refer to the evaluation guidelines of VLMEvalKit

<!-- <img src="assets/radar_v1.png" width="400" /> -->

πŸ† MMSI-Bench Leaderboard

ModelAvg. (%)Type
πŸ₯‡ Human Level97.2Baseline
πŸ₯ˆ o341.0Proprietary
πŸ₯‰ GPT-4.540.3Proprietary
Gemini-2.5-Pro--Thinking37.0Proprietary
Gemini-2.5-Pro36.9Proprietary
Doubao-1.5-pro33.0Proprietary
GPT-4.130.9Proprietary
Qwen2.5-VL-72B30.7Open-source
NVILA-15B30.5Open-source
GPT-4o30.3Proprietary
Claude-3.7-Sonnet--Thinking30.2Proprietary
Seed1.5-VL29.7Proprietary
InternVL2.5-2B29.0Open-source
InternVL2.5-8B28.7Open-source
DeepSeek-VL2-Small28.6Open-source
InternVL3-78B28.5Open-source
InternVL2.5-78B28.5Open-source
LLaVA-OneVision-72B28.4Open-source
NVILA-8B28.1Open-source
InternVL2.5-26B28.0Open-source
DeepSeek-VL227.1Open-source
InternVL3-1B27.0Open-source
InternVL3-9B26.7Open-source
Qwen2.5-VL-3B26.5Open-source
InternVL2.5-1B26.1Open-source
InternVL2.5-4B26.3Open-source
Qwen2.5-VL-7B25.9Open-source
InternVL3-8B25.7Open-source
Llama-3.2-11B-Vision25.4Open-source
InternVL3-2B25.3Open-source
πŸƒ Random Guessing25.0Baseline
LLaVA-OneVision-7B24.5Open-source
DeepSeek-VL2-Tiny24.0Open-source
Blind GPT-4o22.7Baseline

Acknowledgment

MMSI-Bench makes use of data from existing image datasets: ScanNet, nuScenes, Matterport3D, Ego4D, AgiBot-World, DTU, DAVIS-2017 ,and Waymo. We thank these teams for their open-source contributions.

Contact

  • β€”Sihan Yang: sihany077@gmail.com
  • β€”Runsen Xu: runsxu@gmail.com

Citation

bibtex
@article{yang2025mmsi,
  title={MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence},
  author={Yang, Sihan and Xu, Runsen and Xie, Yiman and Yang, Sizhe and Li, Mo and Lin, Jingli and Zhu, Chenming and Chen, Xiaochen and Duan, Haodong and Yue, Xiangyu and Lin, Dahua and Wang, Tai and Pang, Jiangmiao},
  journal={arXiv preprint arXiv:2505.23764},
  year={2025}
}