Fysics-AI/OmniPhysics-Caption_benchmark

OmniFysics-Captioner: Grounding Omni-Modal Understanding in the Physical World for Better Captioning

<p align="center"> 🌐 <a href="https://github.com/Fysics-AI/OmniFysics-Captioner"> Project</a> • 📈 <a href="#benchmark-overview"> Benchmark Overview</a> • 🧪 <a href="#what-opc-measures"> What OPC Measures</a> • 📊 <a href="#daily-physics-50k-subset"> Daily-Physics 50K Subset</a> • 📚 <a href="#citation"> Citation</a> </p>

Introduction

Building omni-modal models with physical intelligence requires benchmarks that test whether generated captions preserve information from both visual and audio streams. Existing detailed-caption benchmarks provide strong visual, event-level, or cloze-based evaluation, but do not always test audiovisual coverage and physical evidence together.

OmniPhysCap (OPC) is an audiovisual caption benchmark designed to evaluate how well generated captions retain omni-modal information from videos. It covers general visual semantics, temporal relations, audio, speech/OCR, and audiovisual alignment, with targeted physics-interaction and physics-outcome questions added to diagnose physical understanding. It is omission-aware: each question includes an explicit Not Mentioned option.

<p align="center"> <img src="fig/case-study.png" alt="Qualitative physical-perception case study" width="100%"> </p>

Contents

• —Benchmark Overview
• —What OPC Measures
• —Daily-Physics 50K Subset
• —Citation
• —License

Benchmark Overview

OPC contains 1,000 audiovisual clips of 6–60 seconds and 8,000 multiple-choice probes, with 5–10 probes assigned per video. The probes primarily evaluate audiovisual caption coverage across general semantics, temporal relations, audio, speech/OCR, and audiovisual alignment; physics interaction and physics outcome are targeted subsets rather than the entire benchmark.

Each question includes an explicit Not Mentioned option, distinguishing omitted evidence from conflicting evidence in free-form captions.

The benchmark video data is released as MP4 files in media/eval/:

[🔗 Fysics-AI/OmniPhysics-Caption_benchmark/tree/main/media/eval](https://huggingface.co/datasets/Fysics-AI/OmniPhysics-Caption_benchmark/tree/main/media/eval)

What OPC Measures

OPC is an omission-aware audiovisual caption diagnostic that evaluates whether a generated caption preserves visual, temporal, audio, speech/OCR, and cross-modal evidence. Its targeted physics questions additionally probe object interactions, material responses, state changes, and physical outcomes. Machine-assisted construction, blind full-video verification, and the explicit Not Mentioned option distinguish omitted evidence from conflicting evidence.

<p align="center"> <img src="fig/opc-benchmark-results.png" alt="OPC benchmark results" width="360"> </p>

The official evaluation scripts are released alongside the project:

[🧪 Fysics-AI/OmniFysics-Captioner/tree/main/evaluation](https://github.com/Fysics-AI/OmniFysics-Captioner/tree/main/evaluation)

Daily-Physics 50K Subset

Daily-Physics 50K is a physics-aware audiovisual corpus for detailed video-caption training. It covers six top-level physical-event categories and 23 observable event subcategories, ranging from brief local interactions to extended multi-stage processes. The corpus is constructed from heterogeneous video sources through event-query retrieval, clip construction, media-integrity checks, content-quality screening, and deduplication.

<p align="center"> <img src="fig/daily-physics-overview.png" alt="Daily-Physics 50K overview and data distribution" width="100%"> </p>

This repository releases a 1,000-video open subset of Daily-Physics 50K for download:

[🔗 Fysics-AI/OmniPhysics-Caption_benchmark/tree/main/media/train](https://huggingface.co/datasets/Fysics-AI/OmniPhysics-Caption_benchmark/tree/main/media/train)

The complete Daily-Physics 50K release will be linked here when available.

Citation

@article{qiu2026omnifysicscaptioner,
  title   = {OmniFysics-Captioner: Grounding Omni-Modal Understanding in the Physical World for Better Captioning},
  author  = {Qiu, Kaixiang and Han, Minghao and Liu, Keliang and Liu, Yizhou and Han, Jinghang and Jiang, Yue and Wang, Shunli and Zhang, Lihua and Yang, Dingkang},
  journal = {arXiv preprint},
  year    = {2026}
}

License

The content of this repository is released under the Apache License 2.0 with an additional non-commercial restriction: it may be used, reproduced, and distributed for research and educational purposes only. Any commercial use is prohibited without prior written permission from the maintainers. Source videos remain subject to the licenses of their original datasets.