lucky-lance/OmniInteract

OmniInteract

Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants

OmniInteract is a streaming benchmark for real-time omnimodal LLMs, evaluated through their native online inference over continuous audio-visual streams. User queries and ambient sounds live in the audio track, visual events live in the video, and a model must decide whether, when, and what to respond — without lookahead to future content.

• —📄 Paper: arXiv:2605.26485
• —💻 Code & evaluation pipeline: github.com/Lucky-Lance/OmniInteract

Dataset at a glance

• —250 videos, 1,430 temporally grounded response slots
• —1Q1A (210 videos / 1,062 slots): localized single-response interaction — real-time, proactive, and nested
• —1QnA (40 videos / 368 slots): long-horizon continuous task monitoring (one instruction → many time-grounded answers)
• —Domains: Chinese daily-life interaction (home, gym, museum, shopping, …) and English mathematical reasoning

Download

The dataset ships as a single archive data.tar.gz. Extract it to obtain the structure below:

tar -xzf data.tar.gz

Structure

.
├── 1q1a/                       # Chinese daily-life QA, 150 videos
│   ├── videos/0001.mp4 ... 0150.mp4
│   ├── annotations/0001.json ... 0150.json
│   └── video_json_map.json
├── 1q1a_math/                  # English math reasoning, 60 videos
│   ├── videos/0001.mp4 ... 0060.mp4
│   ├── annotations/0001.json ... 0060.json
│   └── video_json_map.json
└── 1qna/                       # Long-horizon cooking task monitoring, 40 videos
    ├── videos_bench/
    │   ├── captaincook4d/*.mp4
    │   └── egoper/*.mp4
    └── annotations/
        ├── captaincook4d/*.json
        └── egoper/*.json

1q1a (150) + 1q1a_math (60) = 210 videos form the 1Q1A split; 1qna (40) forms the 1QnA split. For each video NNNN.mp4, its ground truth is the same-numbered NNNN.json.

Annotation schema

Each 1q1a / 1q1a_math annotation is a list of QA entries:

[
  {
    "question_time": "00:01",
    "question_text": "...",
    "answer_time": "00:36",
    "answer_text": "...",
    "question_type": "realtime",
    "is_interrupted": false
  }
]

1q1a_math additionally carries a "source" URL per entry pointing to the original math problem.

video_json_map.json

Each 1Q1A subset ships a map with per-video metadata:

{
  "total": 150,
  "entries": [
    {"video": "videos/0001.mp4", "annotation": "annotations/0001.json", "scene_type": "nested"},
    {"video": "videos/0091.mp4", "annotation": "annotations/0091.json", "scene_type": "multi_turn"}
  ]
}

scene_type is "nested" for the 60 videos probing context switch / resumption, and "multi_turn" otherwise.

Usage

Evaluation (data preparation, LLM judging, slot scoring, and the paper-table metrics IA-QTF1 / IDS / NCCS) is reproduced by the code repository: github.com/Lucky-Lance/OmniInteract.

Citation

@article{lu2026omniinteract,
  title   = {OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants},
  author  = {Lu, Xudong and Li, Xueying and Wang, Annan and Bo, Yang and Chen, Jinpeng and Li, Zengliang and Yang, Nianzu and Liu, Rui and Yang, Xue and Hou, Jingwen and Li, Hongsheng},
  journal = {arXiv preprint arXiv:2605.26485},
  year    = {2026}
}