zai-org/MotionBench

MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models

<font size=5><div align='center' > [🍎 Project Page] [📖 arXiv Paper] [📊 Dataset] [💻 GitHub] [🏆 Leaderboard] [🏆 HF Leaderboard] </div></font>

<p align="center"> <img src="https://raw.githubusercontent.com/THUDM/MotionBench/main/docs/image1.png" width="96%" height="50%"> </p>

MotionBench is a comprehensive evaluation benchmark designed to assess the fine-grained motion comprehension of video understanding models. It evaluates models' motion-level perception through six primary categories of motion-oriented question types and includes data collected from diverse sources, ensuring a broad representation of real-world video content.

🔥 News

• —`2025.02.27` 🎉🎉🎉 MotionBench is accepted by CVPR 2025!!
• —`2025.01.06` 🌟🌟🌟 We released MotionBench, a new benchmark for fine-grained motion comprehension!

Introduction

In recent years, vision language models (VLMs) have made significant advancements in video understanding. However, a crucial capability — fine-grained motion comprehension — remains under-explored in current benchmarks. To address this gap, we propose MotionBench, a comprehensive evaluation benchmark designed to assess the fine-grained motion comprehension of video understanding models.

Features

1. 1.Core Capabilities: Six core capabilities for fine-grained motion understanding, enabling the evaluation of motion-level perception.
2. 2.Diverse Data: MotionBench collects diverse video from the web, public datasets, and self-synthetic videos generated via Unity3, capturing a broad distribution of real-world application.
3. 3.High-Quality Annotations: Reliable benchmark with meticulous human annotation and multi-stage quality control processes.

<p align="center"> <img src="https://raw.githubusercontent.com/THUDM/MotionBench/main/docs/image2.png" width="50%" height="20%"> </p>

Dataset

License

Our dataset is under the CC-BY-NC-SA-4.0 license.

MotionBench is only used for academic research. Commercial use in any form is prohibited. We do not own the copyright of any raw video files. If there is any infringement in MotionBench, please contact shiyu.huang@aminer.cn or directly raise an issue, and we will remove it immediately.

Download and Usage

To use the benchmark, you should download video_info.meta.jsonl from this repository.

Install video2dataset first:

pip install video2dataset
pip uninstall transformer-engine

Caption dataset

We release a dataset of 5,000 videos with manually annotated fine-grained motion descriptions, which are annotated and double-checked together with the benchmark annotation process. Each video includes dynamic information descriptions with annotation density reaching 12.63 words per second.

Publically available dataset

For publically available data (MedVid, SportsSloMo, HA-ViD), we do not provide the original video files. Please refer to the GitHub repository for the mapping files and instructions on how to prepare the video clips.

Install MotionBench

pip install -e .

Evaluation and Submission

To prepare evaluation results and submit to the leaderboard:

cd scripts
python test_acc.py

After execution, you will get an evaluation results file random_answers.json in the scripts directory. You can submit the results to the leaderboard.

Citation

If you find our work helpful for your research, please consider citing our work.

@misc{hong2024motionbench,
      title={MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models}, 
      author={Wenyi Hong and Yean Cheng and Zhuoyi Yang and Weihan Wang and Lefan Wang and Xiaotao Gu and Shiyu Huang and Yuxiao Dong and Jie Tang},
      year={2024},
      eprint={2501.02955},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}