Datasetpublic

Video-Reason/VBVR-MultiStep-Bench

VBVR-MultiStep-Bench The frozen 180-instance public evaluation split released alongside the VBVR-MultiStep training corpus. Designed for long-horizon multi-step image-to-video (I2V) reasoning evaluation. This dataset is part of the VBVR (Very Big Video Reasoning Suite) project. See the parent suite at https://video-reason.com and the suite paper VBVR: A Very Big Video Reasoning Suite (Wang et al., ICML 2026). At a glance Property Value Tasks 36… See the full description on the dataset page: https://huggingface.co/datasets/Video-Reason/VBVR-MultiStep-Bench.

sourceHugging Facecc-by-4.0updated 5mo agoView on Hugging Face

0likes167downloads

Dataset Card

VBVR-MultiStep-Bench

The frozen 180-instance public evaluation split released alongside the VBVR-MultiStep training corpus. Designed for long-horizon multi-step image-to-video (I2V) reasoning evaluation.

This dataset is part of the VBVR (Very Big Video Reasoning Suite) project. See the parent suite at <https://video-reason.com> and the suite paper VBVR: A Very Big Video Reasoning Suite (Wang et al., ICML 2026).

At a glance

Property	Value
Tasks	36 parameterized tasks (`Multi-01` … `Multi-36`)
Reasoning families	Navigation, Planning, CSP, Execution, Geometry, Physics
Instances	180 (5 per task × 36)
Per-instance artifacts	5 (see below)
License	CC-BY-4.0

Five-artifact data contract

Every instance lives at:

Multi-XX_<name>_data-generator/Multi-XX_<name>_data-generator_task/Multi-XX_<name>_data-generator_<id>/

and contains exactly:

File	Role
`first_frame.png`	Model conditioning image (the only visual input the model receives at inference)
`prompt.txt`	Natural-language task contract
`final_frame.png`	Target endpoint (held out from the model)
`ground_truth.mp4`	Reference rollout demonstrating the correct trajectory
`question_metadata.json`	Seed, version, tolerances, task-specific fields

A top-level metadata.parquet indexes every instance with the task id, family, seed, and per-instance metadata for fast filtering.

Reasoning families

Family	Characteristic	Released tasks
Navigation	Discrete motion under adjacency / obstacle constraints	6
Planning	Operator-based state transformation	6
CSP	Incremental labeling under global consistency	6 (3 used for human judging)
Execution	Clocked deterministic update rules	6
Geometry	Ordered constructive geometry	6
Physics	Continuous dynamics with contact / conservation	6

Tasks Multi-13, Multi-14, Multi-15 (CSP) are excluded from the human-judging pool described in the paper but are included in this release for completeness.

Intended use

—Primary use: trajectory-level evaluation of I2V systems under a fixed five-artifact contract.
—Comparison protocol: blind human pairwise judging on three independent axes — process correctness, reference fidelity, render quality.
—Companion training corpus: Video-Reason/VBVR-MultiStep (~360k samples).

Loading

python

import pandas as pd
meta = pd.read_parquet("hf://datasets/Video-Reason/VBVR-MultiStep-Bench/metadata.parquet")

Or pull a single instance:

python

from huggingface_hub import hf_hub_download
prompt_path = hf_hub_download(
    "Video-Reason/VBVR-MultiStep-Bench",
    "Multi-01_maze_shortest_path_data-generator/Multi-01_maze_shortest_path_data-generator_task/Multi-01_maze_shortest_path_data-generator_00000000/prompt.txt",
    repo_type="dataset",
)

License

Released under CC-BY-4.0. The reference rollouts are produced from generators that consume only released task definitions; no third-party copyrighted content is embedded.

Wan2.2-I2V-A14B (Apache-2.0) is referenced as a baseline model and a fine-tuning ancestor for VBVR-Wan2.2; this dataset does not redistribute Wan2.2 weights.

Responsible AI

This dataset is fully synthetic — generators produce every instance from controlled parameters. There are no human subjects, no scraped media, and no personal information. See the Croissant file for the complete RAI metadata.