shuzhig/VideoHallucer

VideoHallucer (mirror)

A redistribution of the VideoHallucer benchmark, bundled together with its videos so the whole benchmark comes down in a single snapshot_download.

This is not the official release. All credit goes to the original authors. Official code: https://github.com/patrick-tssn/VideoHallucer · Official data: https://huggingface.co/datasets/bigai-nlco/VideoHallucer

Dataset description

• —Repository: VideoHallucer
• —Paper: 2406.16338
• —Leaderboard: https://videohallucer.github.io/
• —Point of contact: Yuxuan Wang

VideoHallucer is the first comprehensive benchmark for hallucination detection in large video-language models (LVLMs). It categorizes hallucinations into intrinsic and extrinsic types, with subcategories for object-relation, temporal, semantic detail, external factual, and external non-factual hallucination.

Evaluation is adversarial binary VideoQA: every item is a pair of yes/no questions over the same video — a basic question whose answer is grounded in the video, and a hallucination question crafted so a hallucinating model answers it wrongly. A model is only credited when it gets both right, which defeats the trivial always-yes / always-no strategies that binary QA otherwise rewards.

Data statistics

Contents

object_relation/     object_relation.json     + videos/
temporal/            temporal.json            + videos/
semantic_detail/     semantic_detail.json     + videos/
external_factual/    external_factual.json    + videos/
external_nonfactual/ external_nonfactual.json + videos/
fact_detect/         fact_detect.json, fact_detect_yn.json, modify.py + videos/
interaction/         interaction.json         + videos/

Each subset directory carries its own videos/ folder, and the video field in each JSON is a bare filename resolved against that sibling folder. The external_factual and external_nonfactual subsets are posed over the same video pool, so their videos/ folders overlap.

interaction/ is an extra subset present in the upstream data drop; it is not part of the five headline categories above and is not exposed as a datasets config.

Record format

Every record pairs two questions over one video:

{
  "basic": {
    "video": "1052_6143391925_916_970.mp4",
    "question": "Is there a baby in the video?",
    "answer": "yes"
  },
  "hallucination": {
    "video": "1052_6143391925_916_970.mp4",
    "question": "Is there a doll in the video?",
    "answer": "no"
  },
  "type": "subject"
}

Usage

from huggingface_hub import snapshot_download

root = snapshot_download(repo_id="shuzhig/VideoHallucer", repo_type="dataset")

A single subset:

snapshot_download(
    repo_id="shuzhig/VideoHallucer",
    repo_type="dataset",
    allow_patterns="temporal/*",
)

Annotations only, no videos:

snapshot_download(
    repo_id="shuzhig/VideoHallucer",
    repo_type="dataset",
    allow_patterns="*/*.json",
)

Evaluation

Use the official VideoHallucerKit. The directory layout here matches what the kit expects, so pointing it at the downloaded snapshot root is enough.

Provenance and licensing

Mirrored from `bigai-nlco/VideoHallucer`, released under the MIT license. Source videos are drawn from existing public video corpora and remain subject to their original terms. This mirror asserts no additional rights. If you are an author and would like it removed, please open a discussion.

Citation

@article{wang2024videohallucer,
  title   = {VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Models},
  author  = {Wang, Yuxuan and Wang, Yueqian and Zhao, Dongyan and Xie, Cihang and Zheng, Zilong},
  journal = {arXiv preprint arXiv:2406.16338},
  year    = {2024}
}