ziweix/Video_Reality_Test
Video Reality Test: Can AI-Generated ASMR Videos fool VLMs and Humans? This repository serves as a benchmark for evaluating the realism of video generation models. It specifically focuses on ASMR content, which requires high fidelity in texture rendering, micro-movements, and audio-visual synchronization. Benchmark Structure This benchmark is divided into two difficulty levels. All data is provided in the test split to reflect its purpose for evaluation:… See the full description on the dataset page: https://huggingface.co/datasets/ziweix/Video_Reality_Test.
<p align="center"> <h1 align="center"> Video Reality Test: Can AI-Generated ASMR Videos fool VLMs and Humans?</h1> </p>
<p align="center"> <a href="https://arxiv.org/abs/2512.13281" target="blank"><img src="https://img.shields.io/badge/arXiv-2512.13281-red"></a> <a href="https://video-reality-test.github.io/" target="blank"><img src="https://img.shields.io/badge/Project-Page-brightgreen"></a> <a href="https://huggingface.co/datasets/kolerk/VideoRealityTest" target="blank"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Dataset-blue"></a> <a href="https://github.com/video-reality-test/video-reality-test" target="blank"><img src="https://img.shields.io/badge/Github-Code-orange"></a> </p>
This repository serves as a benchmark for evaluating the realism of video generation models. It specifically focuses on ASMR content, which requires high fidelity in texture rendering, micro-movements, and audio-visual synchronization.
Benchmark Structure
This benchmark is divided into two difficulty levels. All data is provided in the `test` split to reflect its purpose for evaluation:
- `real_hard`: 100 samples. The hard benchmark set featuring complex ASMR scenarios and complex interactions.
- `real_easy`: 49 samples. The easy benchmark set featuring simple ASMR scenarios and simple interactions. ---
How to Use the Benchmark
You can load specific subsets directly using the Hugging Face datasets library. Since this is a benchmark, the data is accessed via the `test` split.
1. Evaluate on the Hard Set (real_hard)
from datasets import load_dataset
# Load the "real" benchmark subset
benchmark = load_dataset("kolerk/Video_Reality_Test", "real_hard")
# Access the first test sample
sample = benchmark['test'][0]
print(sample['video'])
print(sample['reference_image'])
print(sample['prompt'])2. Evaluate on the Easy Set (real_hard)
from datasets import load_dataset
# Load the "real_easy" benchmark subset
benchmark = load_dataset("kolerk/Video_Reality_Test", "real_easy")
# Access the test split
print(f"Total test samples: {len(benchmark['test'])}")