anonymous-video-benchmark/toc_bench

TOC-Bench: A Temporal Object Consistency Benchmark for Video Large Language Models

TOC-Bench is a diagnostic benchmark for evaluating whether Video Large Language Models maintain object identity, state, persistence, and temporal relations throughout a video. It focuses on object-centric phenomena including occlusion, disappearance, reappearance, repeated events, event order, temporal location, duration, conditional state, and relative movement.

Anonymous-review notice. This artifact accompanies a paper currently under double-blind review. Author-identifying paper, repository, and citation links will be added after the review period.

Benchmark overview

Each QA item is associated with an object track and a structured temporal event timeline during construction. The released questions use deterministic answer formats so that predictions can be scored without an LLM judge.

Construction and verification

The benchmark is produced in three stages:

1. 1.Candidate object tracks and temporal events are constructed from the source videos.
2. 2.Structured reasoning units and QA skeletons fix the queried object, event, answer, and distractors before surface realization.
3. 3.Text-only, single-frame, and frame-shuffled filters remove questions that do not require the intended visual or temporal evidence.

The filtering process reduces 45,527 generated items to 17,900 candidates. Five expert annotators then inspect 3,000 sampled candidates in disjoint 600-item assignments. They revise minor wording issues and reject questions with insufficient visual evidence or tracking, event, or answer-label problems, leaving 2,323 released QA pairs.

Data composition

Source videos

The source videos remain governed by their original terms; this repository does not relicense third-party media.

Diagnostic dimensions

Task formats and serialization labels

TOC-Bench has four conceptual task formats: multiple choice, statement pair, numerical count, and event ordering. Three- and four-event ordering share the same task format but use separate serialization labels.

Files and schema

Important fields include:

• —qa_id: unique QA identifier;
• —video_id: identifier of the associated video;
• —video_path: relative media path or source-access reference;
• —format: mcq_4, sp, numerical, ordering_3, or ordering_4;
• —question: natural-language question;
• —option_A–option_D or statement_A–statement_B: answer candidates;
• —events: labeled events for ordering questions;
• —correct_answer or correct_order: deterministic ground truth;
• —metadata.dim: diagnostic dimension;
• —metadata.tier: reasoning tier;
• —metadata.hallucination: hallucination-aware variant, when applicable.

Loading the QA split

from datasets import load_dataset

dataset = load_dataset(
    "anonymous-video-benchmark/toc_bench",
    split="test",
)

print(dataset)
print(dataset[0])

The Dataset Viewer should display 2,323 QA rows. The separate number of unique source videos is 1,951.

Evaluation protocol

Most reported models receive 32 uniformly sampled frames. Model-specific exceptions use the frame budgets explicitly reported in the paper and evaluation configuration.

python eval_runner.py \
  --bench answer_final.json \
  --videos-root videos \
  --model <MODEL_KEY> \
  --frames 32 \
  --out predictions.jsonl

python compute_metrics.py \
  --bench answer_final.json \
  --preds predictions.jsonl \
  --save-summary summary.json

The inference runner uses fixed format-specific prompts and produces resumable JSONL predictions. Scoring uses deterministic answer extraction and exact matching.

Intended use

TOC-Bench is intended for non-commercial academic evaluation of temporal object consistency in Video-LLMs. It is not intended for training identity-recognition systems, surveillance, biometric inference, demographic profiling, or safety-critical deployment decisions.

Limitations

• —Candidate construction depends partly on automatic object and event tools.
• —Uniform frame sampling may miss very brief events.
• —The benchmark emphasizes short-to-medium videos and object-centric temporal reasoning rather than all aspects of video understanding.
• —Results are behavioral evidence and should not be interpreted as direct evidence about a model's internal mechanism.
• —The source datasets introduce their own domain, activity, scene, and geographic biases.

Citation

Citation information will be added after the double-blind review period.