rbler/MMSI-Video-Bench

MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligence

**🌐 Homepage** | **📑 Paper** | **📖 Code** </div>

<!-- contents with emoji -->

🔔 News

🔥[2025-12]: Our MMSI-Video-Bench has been integrated into VLMEvalKit. 🔥[2025-12]: We released our paper, benchmark, and evaluation codes.

📊 Data Details

All of our data is available on Hugging Face and includes the following components:

🎥 Video Data (videos.zip): Contains the video clip file (.mp4) corresponding to each sample. This file is generally not required for most models.

🎥 Frame Data (frames.zip): Contains the frames (.jpg) extracted from each sample's video at the base sampling rate. This rate ensures no key information loss during sampling. Each frame file is named using the format {timestamp}_frame_{base_interval}_{image_id} (e.g., 00:06.00frame1.50_4), where the timestamp, also shown on the top-left corner of the frame, indicates its capture time in the original recording.

🖼️ Reference Image Data (ref_images.zip): Contains the auxiliary images referenced in the questions for each sample.

📝 Text Annotation (mmsivideo.json)：This file contains the annotation information for MMSI-Video-Bench. All time references in the questions correspond to the capture time in the original recording and align with the timestamp flag on each frame. Key fields include:

{
  "ref_images": [Paths to auxiliary images referenced in the question,...],
  "video_list": [
    {
      "path": Video clip file path,
      "start": Timestamp (in seconds) of the first frame of the video clip in the original recording,
      "end": Timestamp (in seconds) of the last frame of the video clip in the original recording,
      "base_fps": Base sampling rate
    },
    ...
  ],
  "frames_list": [[Paths to frames sampled at the base sampling rate,...],...],
  "system_prompt": "...",
  "task_prompt": Task-specific prompt,
  "user_prompt": Question text, with <video> as a placeholder for video and <image> for auxiliary images,
  "format_prompt": Output format requirements,
  "ground_truth": Correct answer,
  "hint": The rationale used when reviewing annotation questions for acceptance, which can serve as a reference prompt if the problem is too difficult.
}

Unless otherwise specified, the model input generally consists of: system_prompt + task_prompt + user_prompt + format_prompt.

🚀 Evaluation

Please refer to the evaluation guidelines in our github repo.

🏆 Leaderboard

<details> <summary>📦 Uniform-50 Setting</summary>

</details>

<details> <summary>📦 Sufficient-Coverage Setting</summary>

</details>

<details> <summary>🤖 Robot Sub-bench</summary>

</details>

<details> <summary>🏠 Indoor Scene Perception Sub-bench</summary>

</details>

<details> <summary>📍 Grounding Sub-bench</summary>

</details>

Note: For the three sub-benchmarks, we take the higher score of each model across the two settings for easier presentation.