foragi/Omni-DuplexEval-Full

Omni-DuplexEval

Omni-DuplexEval is a benchmark for evaluating real-time duplex multimodal interaction. Unlike conventional offline video understanding benchmarks, Omni-DuplexEval focuses on streaming settings where models must continuously process evolving multimodal inputs and decide what to respond and when to respond.

The benchmark contains two scenarios:

• —Real-Time Description (RTD): evaluates continuous streaming description ability.
• —Proactive Reminder (PR): evaluates event-aware and proactive interaction ability.

Omni-DuplexEval contains 660 samples in total, including 300 samples for Real-Time Description and 360 samples for Proactive Reminder. The dataset consists of 9 tasks with human-curated timestamp-level annotations.

Benchmark Structure

Real-Time Description (RTD)

RTD evaluates whether models can continuously describe evolving video content under streaming settings.

Included tasks:

• —RTD_Omni
• —RTD_counting
• —RTDfinegrained_movement
• —RTDinteractionrelation
• —RTD_OCR
• —RTDworldknowledge

Proactive Reminder (PR)

PR evaluates whether models can identify relevant events and respond at the appropriate time.

Included tasks:

• —PReventreminder
• —PRpostevent_reminder
• —PR_correction

Data Format

Each sample contains the following fields:

Annotation Details

Question Audio

question_audio is aligned with the full video duration.

• —For RTD tasks, the question is asked at the beginning of the video.
• —For PR tasks, the question may occur at arbitrary timestamps.

Reference Answers

• —answer1 and answer2 provide human-annotated reference responses.
• —In correction tasks, answer1 additionally contains the corrected target response.
• —In PReventreminder and PRpostevent_reminder, the answer fields are empty.

Reminder Timestamps

• —reminder1 and reminder2 store timestamp annotations related to reminder events.
• —In correction tasks, reminder1 contains the timestamp of the incorrect user statement/event.
• —In RTD tasks, reminder fields are empty.

Data Collection and Safety

Videos are collected from publicly accessible platforms such as YouTube and Bilibili. The dataset does not contain personal sensitive information.

During dataset construction:

• —RTD videos are selected to contain clear temporal dynamics and continuously evolving subjects.
• —PR videos are selected to contain explicit and unambiguous events for stable evaluation.

All samples undergo manual inspection, and potentially unsafe or high-risk content is excluded.