min1321/extreme-when-bench

ExtremeWhenBench

Hour-scale natural-language temporal grounding benchmark.

<p align="center"> <img src="./assets/teaser.png" alt="Short-clip vs hour-long temporal grounding" width="640"> </p>

2,273 open-form natural-language questions over 194 hour-long videos (mean 75.7 min, max 9 hr) sourced from LVBench, MLVU, and VideoMME. The median GT event is 9 s — matched to Charades-STA's 7.1 s — so the same event grain now sits inside a search space ~153× larger.

Companion to Natural-Language Temporal Grounding in Hour-Long Videos is a Search Problem: A Benchmark and Empirical Decomposition — arXiv:2606.12300. Code repository: <https://github.com/naver-ai/ExtremeWhenBench>.

Status

Annotations available now. The reference evaluation script and the lmms-eval task plugin will be released at the GitHub repository linked above together with the camera-ready version of the paper.

Introduction

At hour-scale, the binding constraint is search, not recognition. Video-LLMs are bottlenecked not by localizing a nearby event, but — given a natural-language query — by finding the right region of a long video. Short-video benchmarks can't distinguish these failure modes; hour-scale grounding makes the distinction central and lets the task decompose into a search stage and a localize stage — structurally the retrieve-then-read split that reshaped open-domain QA.

Results

Best per-model mIoU on Charades-STA vs. ExtremeWhenBench (same task). Open Video-LLMs collapse 5–120×, and a frame-level CLIP retriever overtakes every open Video-LLM under our compute budget:

CLIP outperforms every open Video-LLM on ours while sitting in the middle of the pack on Charades-STA — the order flips when search dominates. A failure taxonomy attributes 85% of Video-LLM errors to search; a retrieve-then-ground hybrid recovers 6.7× over the monolithic Video-LLM.

Scaling frames alone doesn't close the gap — Qwen3.5-9B keeps climbing out to N=2,048 frames but remains far below the retrieval baseline:

<p align="center"> <img src="./assets/figure3_sweep.png" alt="Frame-count sweep, Qwen3.5-9B" width="560"> </p>

Queries are open-form natural language, not template-bound: MATTR 0.78 vs. 0.60 (TVBench) and 0.54 (Charades-STA); 1,578 unique 4-gram stems (~25× per-question gap over TVBench).

Loading

from datasets import load_dataset
ds = load_dataset("min1321/extreme-when-bench", split="test")
print(ds[0])

Schema

Predictions are [start, end] intervals in seconds. Metrics: mIoU, R@0.3 / R@0.5 / R@0.7, parse-failure rate (parse failures count as IoU = 0).

Source videos

This dataset ships annotations only — download the videos from their source corpora:

The source_corpus, youtube_url, and video_duration_s columns give you the per-video lookup.

Evaluation

Two evaluation paths will be released at the companion GitHub repo together with the camera-ready (see Status above):

• —Reference vLLM script — async OpenAI client that streams video URLs to a vLLM serve and parses [start, end] predictions. Matches the paper's Table 4 (Qwen3.5-9B, num_frames=768, enable_thinking=False): mIoU 0.0469.
• —lmms-eval task plugin — drop-in for EvolvingLMMs-Lab/lmms-eval, with a small adapter patch (+49 lines) that lets the openai adapter send video as a URL so vLLM can decode server-side (mIoU 0.0485 on the same setup). We plan to upstream the patch + task as a PR to EvolvingLMMs-Lab/lmms-eval.

Reported numbers we will reproduce:

Citation

@article{seo2026extremewhenbench,
  title   = {Natural-Language Temporal Grounding in Hour-Long Videos
             is a Search Problem: A Benchmark and Empirical Decomposition},
  author  = {Seo, Sukmin and Kim, Geewook},
  journal = {arXiv preprint arXiv:2606.12300},
  year    = {2026}
}

License

Annotations: CC-BY-4.0. Copyright (c) 2026-present NAVER Cloud Corp. Source videos are governed by the licenses of LVBench, MLVU, and VideoMME respectively; this dataset does not redistribute them.