DanBenAmi/HERBench

HERBench: A Benchmark for Multi-Evidence Integration in Video Question Answering

<div align="center">

<img src="assets/HERBench_logo.png" alt="HERBench Logo" width="400"/>

     

A challenging benchmark for evaluating multi-evidence integration capabilities of vision-language models

🎉 HERBench has been accepted to CVPR 2026!

</div>

🆕 New: Lite-v2 config. We released a refreshed `lite_v2` version of the Lite split (1,971 questions / 68 videos) in which 9 of the 12 tasks were regenerated and went through additional manual refinement for higher quality, while TSO, SVA, and RLPC are carried over unchanged from lite. Load it with load_dataset("DanBenAmi/HERBench", "lite_v2"). The original lite config remains available unchanged.

📋 Dataset Summary

HERBench is a challenging benchmark designed to evaluate vision-language models on multi-evidence integration in long videos. Unlike existing benchmarks where questions can often be answered from single frames, HERBench enforces a High Evidential Requirement (ER) where each question requires aggregating at least k ≥ 3 distinct, temporally separated visual cues.

<div align="center"> <img src="assets/Teaser_plot.jpg" alt="HERBench Teaser" width="800"/> </div>

Key Statistics

Lite-v2 uses the same 68 videos as Lite. 9 of the 12 tasks were regenerated with additional manual refinement; TSO, SVA, and RLPC are identical to Lite.

Why HERBench?

Current video QA benchmarks often allow models to answer questions using single frames or limited context, failing to test true multi-evidence reasoning. HERBench addresses this by:

✅ Enforcing multi-evidence integration - Each question requires k ≥ 3 temporally separated frames ✅ Preventing single-frame shortcuts - Questions cannot be answered from isolated frames ✅ Testing compositional reasoning - Combines temporal, spatial, and causal reasoning ✅ Evaluating long-video understanding - Average video length of 6.6 minutes

🎯 Choose Your Version

HERBench is available in two versions to accommodate different storage and computational constraints:

Full Version (~161 GB)

• —27,631 questions across 335 videos
• —Complete benchmark for comprehensive evaluation
• —Recommended for: Final paper results, thorough model evaluation, benchmarking

Lite Version (~35 GB) 🚀

• —2,000 questions across 68 videos (7.2% subset)
• —Same task distribution and difficulty as full version
• —Videos sampled to maintain diversity across all 12 tasks
• —Recommended for: Quick prototyping, limited storage, initial experiments, development

Both versions maintain the same quality standards and high evidential requirements!

📊 Leaderboard

Current state-of-the-art results on HERBench (Full benchmark):

TR&C = Temporal Reasoning & Chronology, R&T = Referring & Tracking, GC&V = Global Consistency & Verification, ME&N = Multi-Entity Aggregation & Numeracy

Key Findings:

• —🔍 Referring & Tracking is easier: Models perform best on R&T tasks (avg. 66.8%) compared to other categories
• —🧩 Multi-evidence is challenging: Overall accuracy of 38.2% shows substantial room for improvement
• —📏 Top performers: Ovis-2.5-9B (42.1%) and InternVL3.5-14B (41.5%) lead the benchmark
• —⚖️ Task variance: Performance varies significantly across task families, with GC&V and ME&N being most challenging

📈 MRFS Analysis

HERBench requires significantly more evidence integration than existing benchmarks, as measured by the Minimum Required Frame-Set (MRFS) metric:

<div align="center"> <img src="assets/mrfs_res.jpg" alt="MRFS Comparison" width="600"/> </div>

Key Insights:

• —HERBench has the highest MRFS (5.49) among video QA benchmarks, requiring integration of ~5.5 frames on average
• —4× larger than existing benchmarks with lower text-only accuracy (less language bias)
• —Higher evidential requirement: Questions cannot be answered from single frames or limited context
• —Demonstrates the need for true multi-evidence reasoning in video understanding

🎯 Dataset Features

High Evidential Requirement (ER)

Each question in HERBench is designed to require:

1. 1.Multiple evidence pieces (k ≥ 3 frames minimum)
2. 2.Temporal separation between evidence frames
3. 3.Compositional reasoning across evidence
4. 4.Integration of visual information from different moments

12 Compositional Task Types

Temporal Reasoning & Chronology

Referring & Tracking

Global Consistency & Verification

Multi-Entity Aggregation & Numeracy

Video Sources

Videos are sourced from diverse, high-quality datasets:

• —WildTrack (56 segments): Multi-camera pedestrian tracking scenes
• —HD-EPIC (176 videos): First-person egocentric daily activities
• —PersonPath22 (24 videos): Person tracking scenarios
• —Movie Trailers (81 videos): Narrative storytelling content

📥 Dataset Structure

HERBench/
├── data/
│   ├── herbench_annotations.json        # Full: 27,631 questions
│   ├── herbench_annotations_lite.json   # Lite: 2,000 questions
│   ├── task_metadata.json               # Task descriptions (shared)
│   ├── video_metadata.json              # Video information (shared)
│   └── README_DATA.md                   # Data format documentation
├── videos/
│   ├── videos.tar.part.00               # Lite videos start here
│   ├── videos.tar.part.01               # |
│   ├── videos.tar.part.02               # | Lite: parts 00-03 (~35GB)
│   ├── videos.tar.part.03               # |
│   ├── videos.tar.part.04               # |
│   ├── ...                              # | Full: all parts 00-XX (~161GB)
│   ├── videos.tar.part.XX               # |
│   ├── videos.tar.checksums.txt         # SHA256 checksums
│   └── videos_lite_info.txt             # Info about archive structure
├── herbench.py                          # HF Hub loading script (powers Dataset Viewer)

Archive Structure: Videos are organized so that Lite videos are in the first archive parts (00-03), and Full-only videos are in the remaining parts. This allows efficient downloading of either version without duplication.

Dataset Viewer: The HF Dataset Viewer uses herbench.py to load and preview the dataset. The script defines a stable schema that handles the varying metadata structures across different task types, ensuring efficient streaming and compatibility with Arrow/Parquet format.

Annotation Format

Each sample contains:

{
  "question_id": "HER_001234",
  "video_id": "cam2_segment_4_180s_240s",
  "video_path": "videos/WildTrack/cam2_segment_4_180s_240s.mp4",
  "question": "What is the main activity happening throughout the video?",
  "choices": [
    "A. People walking across the scene",
    "B. People standing and talking",
    "C. People running in the same direction",
    "D. People sitting on benches",
    "E. People cycling through the area"
  ],
  "answer": "A",
  "answer_index": 0,
  "answer_text": "People walking across the scene",
  "task_type": "activity_recognition",
  "metadata": {
    "source_dataset": "WildTrack",
    "duration": 60.0,
    "resolution": "1920x1080",
    "difficulty": "medium"
  }
}

For detailed format documentation, see data/README_DATA.md.

🚀 Quick Start

1. Download the Dataset

Option A: Using Hugging Face CLI (Recommended)

# Install Hugging Face CLI
pip install huggingface-hub

# Download FULL version (27,631 questions, ~161 GB)
huggingface-cli download DanBenAmi/HERBench --repo-type dataset --local-dir HERBench

# Download LITE version only (2,000 questions, ~35 GB videos)
huggingface-cli download DanBenAmi/HERBench \
    --include "data/herbench_lite.parquet" \
    --include "data/herbench_lite_v2.parquet" \
    --include "data/*metadata.json" \
    --include "videos/videos.tar.part.00" \
    --include "videos/videos.tar.part.01" \
    --include "videos/videos.tar.part.02" \
    --include "videos/videos.tar.part.03" \
    --include "videos/videos_lite_info.txt" \
    --include "videos/videos.tar.checksums.txt" \
    --local-dir HERBench

# Or download only annotations (no videos, ~6 MB)
huggingface-cli download DanBenAmi/HERBench --include "data/*.parquet" --include "data/*metadata.json" --local-dir HERBench

Option B: Using Python (Datasets Library)

The dataset is provided in Parquet format for optimal compatibility with HuggingFace Datasets and reliable schema handling.

from datasets import load_dataset

# Load FULL version (default) - 27,631 questions
dataset_full = load_dataset("DanBenAmi/HERBench", "full")
print(f"Total questions: {len(dataset_full['test'])}")

# Access test split
test_data = dataset_full["test"]

# Get a single example
example = test_data[0]
print(f"Question: {example['question']}")
print(f"Choices: {example['choices']}")
print(f"Answer: {example['answer']}")
print(f"Task: {example['task_type']}")
print(f"Video: {example['video_path']}")

# Load LITE version - 2,000 questions (7.2% sample)
dataset_lite = load_dataset("DanBenAmi/HERBench", "lite")
print(f"Lite questions: {len(dataset_lite['test'])}")

# Load LITE-v2 (refined) - 1,971 questions, same 68 videos
dataset_lite_v2 = load_dataset("DanBenAmi/HERBench", "lite_v2")
print(f"Lite-v2 questions: {len(dataset_lite_v2['test'])}")

Schema: Each example contains:

• —question_id - Unique question identifier
• —video_id - Video identifier
• —video_path - Path to video file
• —question - Question text
• —choices - List of 5 multiple-choice options
• —answer - Correct answer (A/B/C/D/E)
• —answer_index - Zero-indexed answer position (0-4)
• —answer_text - Answer value
• —task_type - Task category name
• —source_dataset - Source dataset name
• —duration - Video duration in seconds (float)
• —resolution - Video resolution (width x height)
• —metadata_json - Full metadata as JSON string

Note: Original JSON files are also available in the data/ folder for users who need the raw format for custom processing.

2. Extract Videos

For Full Version:

cd HERBench/videos

# Concatenate all split archives
cat videos.tar.part.* > videos_full.tar

# Extract videos
tar -xvf videos_full.tar

# Verify checksums (optional)
sha256sum -c videos.tar.checksums.txt

# Clean up tar file (optional)
rm videos_full.tar

For Lite Version:

cd HERBench/videos

# Concatenate only lite archives (parts 00-03)
cat videos.tar.part.{00..03} > videos_lite.tar

# Extract videos
tar -xvf videos_lite.tar

# Clean up tar file (optional)
rm videos_lite.tar

Note: The archive is structured so lite videos are in the first parts (00-03). This means if you download the full version, you automatically have the lite videos too!

Troubleshooting Extraction Issues

If you encounter "This does not look like a tar archive" or other errors:

1. 1.Verify checksums to check download integrity:

   cd HERBench/videos
   sha256sum -c videos.tar.checksums.txt

1. 1.Check file format of downloaded parts:

   file videos.tar.part.00
   # Expected: "POSIX tar archive (GNU)" or similar

1. 1.Test before full extraction:

   cat videos.tar.part.{00..03} > videos_lite.tar
   tar -tvf videos_lite.tar | head -10
   # Should list video files like: HD_EPIC/P01/P01-*.mp4

1. 1.Re-download specific parts if checksums fail:

   huggingface-cli download DanBenAmi/HERBench \
       --repo-type dataset \
       --include "videos/videos.tar.part.00" \
       --local-dir HERBench \
       --force-download

If issues persist, please open an issue on GitHub.

3. Load and Use the Data

from datasets import load_dataset

# Load the dataset (choose version)
dataset = load_dataset("DanBenAmi/HERBench", name="full")  # or name="lite"

# Access a sample
sample = dataset['test'][0]
print(f"Question: {sample['question']}")
print(f"Choices: {sample['choices']}")
print(f"Answer: {sample['answer']}")
print(f"Video: {sample['video_path']}")
print(f"Task: {sample['task_type']}")

# Filter by task type
temporal_questions = [
    q for q in dataset['test']
    if q['task_type'] == 'temporal_reasoning'
]
print(f"Temporal reasoning questions: {len(temporal_questions)}")

# Compare versions
dataset_full = load_dataset("DanBenAmi/HERBench", name="full")
dataset_lite = load_dataset("DanBenAmi/HERBench", name="lite")
print(f"Full: {len(dataset_full['test'])} questions")
print(f"Lite: {len(dataset_lite['test'])} questions")

4. Run Evaluation

# Clone the evaluation code
git clone https://github.com/DanBenAmi/HERBench.git
cd HERBench

# Install dependencies
pip install -r requirements.txt

# Run evaluation on your model
python evaluation/run_evaluation.py \
    model=your_model \
    data_path=./HERBench \
    output_path=./results

📜 Citation

If you use HERBench in your research, please cite:

@article{herbench2025,
  title={HERBench: A Benchmark for Multi-Evidence Integration in Video Question Answering},
  author={Ben-Ami, Dan and Serussi, Gabriele and Cohen, Kobi and Baskin, Chaim},
  journal={arXiv preprint arXiv:2512.14870},
  year={2025}
}

📄 License

This dataset is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0).

Terms of Use

Research Use Only. HERBench is released strictly for non-commercial research and educational purposes. The benchmark is constructed using videos originating from existing datasets and platforms, including WildTrack, HD-EPIC, PersonPath22, and publicly available online videos (e.g., YouTube trailers). All rights to the original video content remain with their respective owners and licensors.

HERBench does not claim ownership of any underlying video content. The use of such materials is intended solely for academic evaluation and analysis, in accordance with the terms of the respective source datasets and platforms.

Removal upon request. If any content owner or rights holder believes that their material has been included in HERBench in a manner that violates applicable terms or rights, please contact us. Upon notification, we will promptly investigate the request and remove the relevant content as appropriate.

🙏 Acknowledgments

We thank the creators of the original video datasets (WildTrack, HD-EPIC, PersonPath22) for making their data publicly available. We also acknowledge the movie studios for releasing promotional trailers.

This work was supported by [Institution/Grant acknowledgments to be added].

📧 Contact

Authors

• —Dan Ben-Ami - danbenami3@gmail.com
• —Gabriele Serussi - serussigabriele@gmail.com
• —Kobi Cohen
• —Chaim Baskin

Support

• —Issues: GitHub Issues
• —Discussions: HF Discussions
• —Email: danbenami3@gmail.com

🔄 Updates

• —v1.0.0 (February 2026): Cleaned dataset — removed 305 questions with corrupted VLM descriptions; Full: 27,631 questions, Lite: 2,000 questions (stratified subsample)

🔗 Links

• —📄 Paper: https://arxiv.org/abs/2512.14870
• —💻 Code: https://github.com/DanBenAmi/HERBench
• —🌐 Project Page: https://gabrieleserussi.github.io/HERBench/
• —🤗 Dataset: https://huggingface.co/datasets/DanBenAmi/HERBench

<div align="center">

Built with ❤️ for advancing video understanding research

If you find HERBench useful, please ⭐ star our [GitHub repository](https://github.com/DanBenAmi/HERBench)!

</div>