VLM-Reasoning/VCR-Bench

1---2dataset_info:3  config_name: vcrbench4  features:5  - name: id6    dtype: int647  - name: video_path8    dtype: string9  - name: duration10    dtype: int6411  - name: dimension12    dtype: string13  - name: multiple-choice14    dtype: bool15  - name: question16    dtype: string17  - name: choices18    struct:19      - name: A20        dtype: string21      - name: B22        dtype: string23      - name: C24        dtype: string25      - name: D26        dtype: string27      - name: E28        dtype: string29  - name: answer30    dtype: string31  - name: reasoning32    list: string33 34configs:35- config_name: vcrbench36  data_files:37  - split: test38    path: v1/videos/meta*39---40 41# VCR-Bench ( A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoning)42 43[**🌐 Homepage**](https://vlm-reasoning.github.io/VCR-Bench/) | [**🤗 Dataset**](https://huggingface.co/datasets/VLM-Reasoning/VCR-Bench) | [**🤗 Paper**](https://huggingface.co/papers/2504.07956) | [**📖 arXiv**](https://arxiv.org/pdf/2504.07956) | [**GitHub**](https://github.com/zhishuifeiqian/VCR-Bench)44 45## Dataset Details46 47As shown in the figure below, current video benchmarks often lack comprehensive annotations of CoT steps, focusing only on the accuracy of final answers during model evaluation while neglecting the quality of the reasoning process. This evaluation approach makes it difficult to comprehensively evaluate model’s actual drawbacks during the CoT reasoning process.48 49<p align="center">50    <img src="https://github.com/vlm-reasoning/VCR-Bench/raw/main/static/images/wrong_steps_case.png" width="80%"> <br>51</p>52 53To fill this gap, we propose <b>VCR-Bench</b>, a benchmark specifically designed to evaluate the <b>V</b>ideo <b>C</b>hain-of-Thought <b>R</b>easoning capabilities of LVLMs.54 55🎯 In VCR-Bench, we have constructed a multi-dimensional evaluation framework, defining **7 distinct task dimensions** that comprehensively cover a diverse range of video types and durations. For each data sample, in addition to providing a standard answer, we have meticulously curated detailed and accurate reference stepwise rationals as CoT annotation.56 57<p align="center">58    <img src="https://github.com/vlm-reasoning/VCR-Bench/raw/main/static/images/cases_dim.png" width="80%"> <br>59</p>60 61To ensure the diversity of video data and the richness of sample information, we curated the VCR-Bench by selecting and integrating data from **14 existing video bench-marks**. These include datasets focused on video perception and comprehension, datasets targeting subject knowledge understanding and reasoning, datasets emphasizing long-form video understanding, datasets specialized in video temporal localization and analysis and datasets dedicated to video scene reasoning.62 63All samples underwent rigorous manual annotation and quality control, ultimately resulting in the creation of VCR-Bench, which includes **859 videos** and **1,034 high-quality question-answer pairs**.64 65<p align="center">66    <img src="https://github.com/zhishuifeiqian/VCR-Bench/raw/main/figs/data.png" width="80%"> <br>67</p>68 69## 🏆 Mini-Leaderboard70We show a mini-leaderboard here and please find more information in our paper or [homepage](https://vlm-reasoning.github.io/VCR-Bench/).71 72| Model                      | Avg |73|----------------------------|:---------:|74| o1| **56.7**  |75| Gemini-2.0-Flash|  51.7 |76| GPT-4o  |52.1|77|GPT4V (low)|46.9|78|Gemini-1.5-Pro|44.0|79|Claude 3.5 Sonnet|41.0|80|Aria-25B|38.2|81|Qwen2.5-VL-72B|37.9|82|LLaVA-Video-72B|36.6|83|LLaVA-OneVision-72B|36.4|84|InternVideo2.5-8B|33.0|85|LLaVA-Video-7B|32.5|86|VideoLLaMA3-7B|32.5|87|InternVL2.5-78B|30.9|88|LLaVA-OneVision-7B|30.7|89|Qwen2.5-VL-7B|30.4|90|MiniCPM-o2.6-8B|26.9|91|InternVL2.5-8B|23.9|92|mPLUG-Owl3-7B|7.3|93|Llama-3.2-11B-Vision|4.9|94 95## ✍️ Citation96If you use our work and are inspired by our work, please consider cite us (available soon):97```98@article{qi2025vcr,99         title={VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoning},100         author={Qi, Yukun and Zhao, Yiming and Zeng, Yu and Bao, Xikun and Huang, Wenxuan and Chen, Lin and Chen, Zehui and Zhao, Jie and Qi, Zhongang and Zhao, Feng},101         journal={arXiv preprint arXiv:2504.07956},102         year={2025}103} 104```