mjuicem/StreamingBench

1---2task_categories:3- question-answering4language:5- en6size_categories:7- 1K<n<10K8 9 10dataset_info:11  - config_name: Real_Time_Visual_Understanding12    features:13      - name: question_id14        dtype: string15      - name: task_type16        dtype: string17      - name: question18        dtype: string19      - name: time_stamp20        dtype: string21      - name: answer22        dtype: string23      - name: options24        dtype: string25      - name: frames_required26        dtype: string27      - name: temporal_clue_type28        dtype: string29    splits:30      - name: Real_Time_Visual_Understanding31        num_examples: 250032 33  - config_name: Sequential_Question_Answering34    features:35      - name: question_id36        dtype: string37      - name: task_type38        dtype: string39      - name: question40        dtype: string41      - name: time_stamp42        dtype: string43      - name: answer44        dtype: string45      - name: options46        dtype: string47      - name: frames_required48        dtype: string49      - name: temporal_clue_type50        dtype: string51    splits:52      - name: Sequential_Question_Answering53        num_examples: 25054 55 56  - config_name: Contextual_Understanding57    features:58      - name: question_id59        dtype: string60      - name: task_type61        dtype: string62      - name: question63        dtype: string64      - name: time_stamp65        dtype: string66      - name: answer67        dtype: string68      - name: options69        dtype: string70      - name: frames_required71        dtype: string72      - name: temporal_clue_type73        dtype: string74    splits:75      - name: Contextual_Understanding76        num_examples: 50077 78  - config_name: Omni_Source_Understanding79    features:80      - name: question_id81        dtype: string82      - name: task_type83        dtype: string84      - name: question85        dtype: string86      - name: time_stamp87        dtype: string88      - name: answer89        dtype: string90      - name: options91        dtype: string92      - name: frames_required93        dtype: string94      - name: temporal_clue_type95        dtype: string96    splits:97      - name: Omni_Source_Understanding98        num_examples: 100099 100    101  - config_name: Proactive_Output102    features:103      - name: question_id104        dtype: string105      - name: task_type106        dtype: string107      - name: question108        dtype: string109      - name: time_stamp110        dtype: string111      - name: ground_truth_time_stamp112        dtype: string113      - name: ground_truth_output114        dtype: string115      - name: frames_required116        dtype: string117      - name: temporal_clue_type118        dtype: string119    splits:120      - name: Proactive_Output121        num_examples: 250122 123 124configs:125  - config_name: Real_Time_Visual_Understanding126    data_files:127    - split: Real_Time_Visual_Understanding128      path: StreamingBench/Real_Time_Visual_Understanding.csv 129      130  - config_name: Sequential_Question_Answering131    data_files:132    - split: Sequential_Question_Answering133      path: StreamingBench/Sequential_Question_Answering.csv134 135  - config_name: Contextual_Understanding136    data_files:137    - split: Contextual_Understanding138      path: StreamingBench/Contextual_Understanding.csv139      140  - config_name: Omni_Source_Understanding141    data_files:142    - split: Omni_Source_Understanding143      path: StreamingBench/Omni_Source_Understanding.csv144      145  - config_name: Proactive_Output146    data_files:147    - split: Proactive_Output148      path: StreamingBench/Proactive_Output_50.csv149    - split: Proactive_Output_250150      path: StreamingBench/Proactive_Output.csv151    152---153# StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding154 155<div align="center">156  <img src="./figs/icon.png" width="100%" alt="StreamingBench Banner">157 158  <div style="margin: 30px 0">159    <a href="https://streamingbench.github.io/" style="margin: 0 10px">🏠 Project Page</a> |160    <a href="https://arxiv.org/abs/2411.03628" style="margin: 0 10px">📄 arXiv Paper</a> |161    <a href="https://huggingface.co/datasets/mjuicem/StreamingBench" style="margin: 0 10px">📦 Dataset</a> |162    <a href="https://streamingbench.github.io/#leaderboard" style="margin: 0 10px">🏅Leaderboard</a>163  </div>164</div>165 166**StreamingBench** evaluates **Multimodal Large Language Models (MLLMs)** in real-time, streaming video understanding tasks. 🌟167 168------169 170[**NEW!** 2025.05.15] 🔥: [Seed1.5-VL](https://github.com/ByteDance-Seed/Seed1.5-VL) achieved ALL model SOTA with a score of 82.80 on the Proactive Output.171 172[**NEW!** 2025.03.17] ⭐: [ViSpeeker](https://arxiv.org/abs/2503.12769) achieved Open-Source SOTA with a score of 61.60 on the Omni-Source Understanding.173 174[**NEW!** 2025.01.14] 🚀: [MiniCPM-o 2.6](https://github.com/OpenBMB/MiniCPM-o) achieved Streaming SOTA with a score of 66.01 on the Overall benchmark.175 176[**NEW!** 2025.01.06] 🏆: [Dispider](https://github.com/Mark12Ding/Dispider) achieved Streaming SOTA with a score of 53.12 on the Overall benchmark.177 178[**NEW!** 2024.12.09] 🎉: [InternLM-XComposer2.5-OmniLive](https://github.com/InternLM/InternLM-XComposer) achieved 73.79 on Real-Time Visual Understanding.179 180------181 182## 🎞️ Overview183 184As MLLMs continue to advance, they remain largely focused on offline video comprehension, where all frames are pre-loaded before making queries. However, this is far from the human ability to process and respond to video streams in real-time, capturing the dynamic nature of multimedia content. To bridge this gap, **StreamingBench** introduces the first comprehensive benchmark for streaming video understanding in MLLMs.185 186### Key Evaluation Aspects187- 🎯 **Real-time Visual Understanding**: Can the model process and respond to visual changes in real-time?188- 🔊 **Omni-source Understanding**: Does the model integrate visual and audio inputs synchronously in real-time video streams?189- 🎬 **Contextual Understanding**: Can the model comprehend the broader context within video streams?190 191### Dataset Statistics192- 📊 **900** diverse videos193- 📝 **4,500** human-annotated QA pairs194- ⏱️ Five questions per video at different timestamps195#### 🎬 Video Categories196<div align="center">197  <img src="./figs/StreamingBench_Video.png" width="80%" alt="Video Categories">198</div>199 200#### 🔍 Task Taxonomy201<div align="center">202  <img src="./figs/task_taxonomy.png" width="80%" alt="Task Taxonomy">203</div>204 205 206## 🔬 Experimental Results207 208### Performance of Various MLLMs on StreamingBench209- All Context210<div align="center">211  <img src="./figs/result_1.png" width="80%" alt="Task Taxonomy">212</div>213 214- 60 seconds of context preceding the query time215<div align="center">216  <img src="./figs/result_2.png" width="80%" alt="Task Taxonomy">217</div>218 219- Comparison of Main Experiment vs. 60 Seconds of Video Context220- <div align="center">221  <img src="./figs/heatmap.png" width="80%" alt="Task Taxonomy">222</div>223 224### Performance of Different MLLMs on the Proactive Output Task225*"≤ xs" means that the answer is considered correct if the actual output time is within x seconds of the ground truth.*226<div align="center">227  <img src="./figs/po.png" width="80%" alt="Task Taxonomy">228</div>229 230 231## 📝 Citation232```bibtex233@article{lin2024streaming,234  title={StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding},235  author={Junming Lin and Zheng Fang and Chi Chen and Zihao Wan and Fuwen Luo and Peng Li and Yang Liu and Maosong Sun},236  journal={arXiv preprint arXiv:2411.03628},237  year={2024}238}239```240 241https://arxiv.org/abs/2411.03628242