minuzero/VideoKR-Eval

VideoKR-Eval

<p align="center"> 📄 <a href="https://arxiv.org/pdf/2606.05259">ArXiv</a> &nbsp｜&nbsp 💻 <a href="https://github.com/Fu-Fu-Fu-Fu/VideoKR">Code</a> &nbsp｜&nbsp 🤗 <a href="https://huggingface.co/collections/minuzero/videokr">Collection</a> </p>

About

This repository contains the VideoKR-Eval benchmark presented in VideoKR: Towards Knowledge- and Reasoning-Intensive Video Understanding (ICML 2026 Spotlight).

VideoKR-Eval is an expert-annotated evaluation benchmark for knowledge- and reasoning-intensive video understanding. Unlike existing benchmarks where a substantial fraction of questions can be answered from a single frame or text alone, VideoKR-Eval ensures that every question requires genuine continuous video understanding and domain knowledge.

Benchmark Construction

VideoKR-Eval is constructed from three existing benchmarks (VideoMMMU, MMVU, SciVideoBench) through a rigorous two-stage process:

1. 1.Multi-Model Single-Frame Filtering: Each example is probed with three frontier models (Qwen3-VL-235B, Claude-4.5-Sonnet, GPT-5.2) using only the question, answer options, and one randomly sampled frame. Only examples that all three models fail to answer correctly in all three trials are retained — yielding 1,254 original examples that genuinely require continuous video understanding.

1. 1.Expert Re-annotation: For the filtered-out examples, domain experts re-annotate new QA pairs using the corresponding videos, ensuring questions are grounded in observable video evidence and require relevant domain knowledge. This produces 746 expert-reannotated examples.

The final benchmark contains 2,000 examples total.

Why VideoKR-Eval?

Existing benchmarks have surprisingly high single-frame answerability rates:

VideoKR-Eval achieves a single-frame answerability rate below 11% for all frontier models, compared to 13–50% on existing benchmarks.

Statistics

• —Total examples: 2,000
• —Original retained examples: 1,254 (from VideoMMMU, MMVU, SciVideoBench)
• —Expert re-annotated examples: 746
• —Disciplines: Natural Sciences, Healthcare, Humanities & Social Sciences, Engineering
• —Subjects: 82 professional subjects

Usage

Evaluation with lmms-eval

cd /path/to/VideoKR/lmms_eval
conda activate videokr_eval

export CUDA_VISIBLE_DEVICES=0
export VIDEOKR_MODEL=/path/to/your-model
export TASKS=videokr_eval
export BATCH_SIZE=1
export RUN_NAME=videokr_eval

bash examples/models/videokr_vllm.sh

Loading the Dataset

from datasets import load_dataset

dataset = load_dataset("minuzero/VideoKR-Eval")

Scoring

Without an API key, the evaluator uses rule-based scoring for multiple-choice questions and skips open-ended questions. To enable a VLM judge for open-ended questions:

export API_TYPE=openai
export OPENAI_API_KEY=...
export OPENAI_BASE_URL=https://api.openai.com/v1
export MODEL_VERSION=gpt-4o

bash examples/models/videokr_vllm.sh

Links

Citation

If you find VideoKR useful in your research, please cite our paper:

@misc{fu2026videokrknowledgereasoningintensivevideo,
      title={VideoKR: Towards Knowledge- and Reasoning-Intensive Video Understanding}, 
      author={Lin Fu and Zheyuan Yang and Yang Wang and Tingyu Song and Arman Cohan and Yilun Zhao},
      year={2026},
      eprint={2606.05259},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2606.05259}, 
}