stephenbasd/MVU-Eval-Data

MVU-Eval Dataset

Paper | Code | Project Page

Dataset Description

The advent of Multimodal Large Language Models (MLLMs) has expanded AI capabilities to visual modalities, yet existing evaluation benchmarks remain limited to single-video understanding, overlooking the critical need for multi-video understanding in real-world scenarios (e.g., sports analytics and autonomous driving). To address this significant gap, we introduce MVU-Eval, the first comprehensive benchmark for evaluating Multi-Video Understanding for MLLMs. Specifically, our MVU-Eval mainly assesses eight core competencies through 1,824 meticulously curated question-answer pairs spanning 4,959 videos from diverse domains, addressing both fundamental perception tasks and high-order reasoning tasks. These capabilities are rigorously aligned with real-world applications such as multi-sensor synthesis in autonomous systems and cross-angle sports analytics. Through extensive evaluation of state-of-the-art open-source and closed-source models, we reveal significant performance discrepancies and limitations in current MLLMs' ability to perform understanding across multiple videos. The benchmark will be made publicly available to foster future research.

🌟 Key Features

• —🎯 First Multi-Video Understanding Benchmark 1,824 QA pairs and 4,959 videos across 8 task categories, bridging perception ↔ reasoning.

• —🧩 Eight Core Competencies Object Recognition (OR), Spatial Understanding (SU), Counting, Comparison, Knowledge-Intensive Reasoning (KIR), In-Context Learning (ICL), Retrieval-Augmented Generation (RAG), and Temporal Reasoning (TR).

• —⚙️ Rigorous Data Pipeline Automated QA generation + dual-round human verification + leakage and utility checks ensure quality and fairness.

• —📊 Comprehensive Evaluation Benchmarked on 30+ open/closed-source MLLMs (e.g., Gemini 2.5 Pro, GPT-4o, Qwen 2.5-VL, InternVL 3), revealing major performance gaps.

🏆 Leaderboard

Category-wise model performance on MVU-Eval. "OR": object recognition. "SU": spatial understanding. "KIR": knowledge-intensive reasoning. "ICL": in-context learning. "RAG": retrieval-augmented generation. "TR": temporal reasoning.

Sample Usage

This section provides a general example of how to evaluate models on the MVU-Eval benchmark using vLLM for inference, as described in the accompanying GitHub repository.

First, download the MVU-Eval dataset and the necessary evaluation scripts.

1. Download Data and Setup Dependencies

# Clone the MVU-Eval dataset, including video files (requires Git LFS)
git lfs install
git clone https://huggingface.co/datasets/MVU-Eval-Team/MVU-Eval-Data /path/to/MVU-Eval-Data

# Download evaluation script and requirements from the Hugging Face Hub
# We rename main_all_MVU_Eval_llama3.py to inference/main.py to align with GitHub instructions
mkdir -p inference
wget https://huggingface.co/datasets/MVU-Eval-Team/MVU-Eval-Data/resolve/main/main_all_MVU_Eval_llama3.py -O inference/main.py
wget https://huggingface.co/datasets/MVU-Eval-Team/MVU-Eval-Data/resolve/main/requirements.py -O requirements.txt

# Install Python packages
pip install -r requirements.txt

# Install ffmpeg for video processing
sudo apt-get update
sudo apt-get install -y ffmpeg

The MVU-Eval QA pairs can be found at: https://huggingface.co/datasets/MVU-Eval-Team/MVU-Eval-Data/resolve/main/MVUEvalQAs.json

2. Start the vLLM Server

This example uses Qwen/Qwen2.5-VL-3B-Instruct. Adjust the model name and resources as needed.

# Start vLLM server (example: Qwen/Qwen2.5-VL-3B-Instruct)
python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-VL-3B-Instruct \
    --served-model-name Qwen/Qwen2.5-VL-3B-Instruct \
    --api-key sk-abc123 \
    --tensor-parallel-size 4 \
    --pipeline-parallel-size 1 \
    --trust-remote-code \
    --dtype auto \
    --gpu-memory-utilization 0.85 \
    --port 8007 \
    --host localhost

Note: Adjust --tensor-parallel-size to your GPU count and memory. If you use another port, update --port in the next step accordingly.

3. Run Inference

Navigate to the inference directory (where main.py was saved) and run the main inference script:

cd inference

# Replace paths/filenames as needed:
python main.py \
  --model_name Qwen/Qwen2.5-VL-3B-Instruct \
  --port 8007 \
  --data_filename QA_json_file.json \
  --data_root /path/to/MVU-Eval-Data/videos \
  --nframes 32 \
  --max_pixels 720

• —--data_filename points to a JSON file (e.g., QA_json_file.json within the dataset directory).
• —--data_root is the root directory containing all videos used in the QA file (e.g., /path/to/MVU-Eval-Data/videos).
• —--nframes (default: 32) is the number of uniformly sampled frames per video.
• —--max_pixels (default: 720) is the max side for frame resizing.

After execution, predictions will be saved under:

inference/Model_output/max_pixel_{max_pixels}_nframes_{nframes}/{QA_json_file_stem}/main/

4. Analyze Results

To generate per-task and overall accuracy tables/plots from the saved predictions, run the analysis script from the inference directory:

python analyze.py

The analysis script will:

• —Aggregate results from Model_output/\u2026/*.json
• —Compute overall and task-wise accuracy
• —Export a markdown table and save comparison plots for reporting

🪶 Citation

If you find MVU-Eval useful for your research, please cite:

@inproceedings{
  peng2025mvueval,
  title={{MVU}-Eval: Towards Multi-Video Understanding Evaluation for Multimodal {LLM}s},
  author={Tianhao Peng and Haochen Wang and Yuanxing Zhang and Zekun Moore Wang and Zili Wang and Ge Zhang and Jian Yang and Shihao Li and Yanghai Wang and Xintao Wang and Houyi Li and Wei Ji and Pengfei Wan and Wenhao Huang and Zhaoxiang Zhang and Jiaheng Liu},
  booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
  year={2025},
  url={https://openreview.net/forum?id=UZD5CQV6f9}
}