shuzhig/elv-halluc-videos

ELV-Halluc — videos + annotations

A self-contained mirror of the ELV-Halluc benchmark (CVPR 2026), bundling the raw .mp4 files together with the annotations so the benchmark can be run without sourcing videos separately.

• —Paper: arXiv:2508.21496
• —Original annotations: HLSv/ELV-Halluc (no videos)
• —Project page: https://elv-halluc.github.io/

This is an unofficial mirror. All credit for the benchmark goes to the original authors; please cite their paper (below) rather than this repo.

Layout

.
├── data/
│   ├── ELV_Halluc.jsonl      # 4,680 binary QA rows
│   └── duration_cache.json   # {filename: duration_seconds}
└── <video_id>.mp4            # 195 videos, flat at the repo root

Videos sit at the repo root, which matches the annotations: the video_path field is a bare filename (mP-pGbYqMt8.mp4), not a path.

Every row has its video, and every video has rows — the annotations and the .mp4 files reference exactly the same set of 195 videos, and duration_cache.json covers the same 195. No filtering or availability checks are needed before evaluating.

Difference from the original benchmark

The original ELV_Halluc.jsonl has 4,800 rows over 200 videos. Five of those videos are no longer retrievable from YouTube — 0YIfDGNOAhE, 90LV7mWt_qU, Atgq7X4pqyk, GlD6cvV-A84, hTUMYdO4PFs all return "Video unavailable" — and they are not distributed by the original dataset repo either. The 120 QA rows depending on them have been removed here, since they cannot be answered from video content.

The removal is a clean subset: it drops whole videos (all 24 of their QA rows), so the design of the benchmark is preserved intact — still 24 QA per video, still an exact 1/3 split across ground_truth / in_video / out_video (1,560 each) and an exact 1/4 split across the four aspects (1,170 each).

Because the row set differs, absolute scores here are not directly comparable to the published leaderboard, which is computed over all 200 videos.

Fields

data/ELV_Halluc.jsonl — one JSON object per line:

Usage

from datasets import load_dataset
from huggingface_hub import snapshot_download

bench = load_dataset("shuzhig/elv-halluc-videos", split="train")   # 4,680 rows
path  = snapshot_download("shuzhig/elv-halluc-videos", repo_type="dataset")

# videos live at the repo root, keyed by the bare filename in `video_path`
video_file = f"{path}/{bench[0]['video_path']}"

Evaluation

The 4,680 rows group into 3,120 adversarial pairs: a ground-truth caption paired with a hallucinated one. A pair counts as correct only when the model answers Yes to the ground-truth question and No to the hallucinated one. Use the original `eval.py`, keeping the row order of ELV_Halluc.jsonl and adding your outputs under a model_response key.

Citation

@misc{lu2025elvhallucbenchmarkingsemanticaggregation,
      title={ELV-Halluc: Benchmarking Semantic Aggregation Hallucinations in Long Video Understanding},
      author={Hao Lu and Jiahao Wang and Yaolun Zhang and Ruohui Wang and Xuanyu Zheng and Yepeng Tang and Dahua Lin and Lewei Lu},
      year={2025},
      eprint={2508.21496},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2508.21496},
}