mjuicem/StreamingBench

StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding

<div align="center"> <img src="./figs/icon.png" width="100%" alt="StreamingBench Banner">

<div style="margin: 30px 0"> <a href="https://streamingbench.github.io/" style="margin: 0 10px">🏠 Project Page</a> | <a href="https://arxiv.org/abs/2411.03628" style="margin: 0 10px">📄 arXiv Paper</a> | <a href="https://huggingface.co/datasets/mjuicem/StreamingBench" style="margin: 0 10px">📦 Dataset</a> | <a href="https://streamingbench.github.io/#leaderboard" style="margin: 0 10px">🏅Leaderboard</a> </div> </div>

StreamingBench evaluates Multimodal Large Language Models (MLLMs) in real-time, streaming video understanding tasks. 🌟

[NEW! 2025.05.15] 🔥: Seed1.5-VL achieved ALL model SOTA with a score of 82.80 on the Proactive Output.

[NEW! 2025.03.17] ⭐: ViSpeeker achieved Open-Source SOTA with a score of 61.60 on the Omni-Source Understanding.

[NEW! 2025.01.14] 🚀: MiniCPM-o 2.6 achieved Streaming SOTA with a score of 66.01 on the Overall benchmark.

[NEW! 2025.01.06] 🏆: Dispider achieved Streaming SOTA with a score of 53.12 on the Overall benchmark.

[NEW! 2024.12.09] 🎉: InternLM-XComposer2.5-OmniLive achieved 73.79 on Real-Time Visual Understanding.

🎞️ Overview

As MLLMs continue to advance, they remain largely focused on offline video comprehension, where all frames are pre-loaded before making queries. However, this is far from the human ability to process and respond to video streams in real-time, capturing the dynamic nature of multimedia content. To bridge this gap, StreamingBench introduces the first comprehensive benchmark for streaming video understanding in MLLMs.

Key Evaluation Aspects

• —🎯 Real-time Visual Understanding: Can the model process and respond to visual changes in real-time?
• —🔊 Omni-source Understanding: Does the model integrate visual and audio inputs synchronously in real-time video streams?
• —🎬 Contextual Understanding: Can the model comprehend the broader context within video streams?

Dataset Statistics

• —📊 900 diverse videos
• —📝 4,500 human-annotated QA pairs
• —⏱️ Five questions per video at different timestamps

🎬 Video Categories

<div align="center"> <img src="./figs/StreamingBench_Video.png" width="80%" alt="Video Categories"> </div>

🔍 Task Taxonomy

<div align="center"> <img src="./figs/task_taxonomy.png" width="80%" alt="Task Taxonomy"> </div>

🔬 Experimental Results

Performance of Various MLLMs on StreamingBench

• —All Context <div align="center"> <img src="./figs/result_1.png" width="80%" alt="Task Taxonomy"> </div>

• —60 seconds of context preceding the query time <div align="center"> <img src="./figs/result_2.png" width="80%" alt="Task Taxonomy"> </div>

• —Comparison of Main Experiment vs. 60 Seconds of Video Context
• —<div align="center"> <img src="./figs/heatmap.png" width="80%" alt="Task Taxonomy"> </div>

Performance of Different MLLMs on the Proactive Output Task

"≤ xs" means that the answer is considered correct if the actual output time is within x seconds of the ground truth. <div align="center"> <img src="./figs/po.png" width="80%" alt="Task Taxonomy"> </div>

📝 Citation

@article{lin2024streaming,
  title={StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding},
  author={Junming Lin and Zheng Fang and Chi Chen and Zihao Wan and Fuwen Luo and Peng Li and Yang Liu and Maosong Sun},
  journal={arXiv preprint arXiv:2411.03628},
  year={2024}
}

https://arxiv.org/abs/2411.03628