Video-Reason/VBVR-MultiStep-Bench

1---2license: cc-by-4.03task_categories:4  - image-to-video5  - text-to-video6language:7  - en8size_categories:9  - n<1K10pretty_name: VBVR-MultiStep-Bench11tags:12  - video-reasoning13  - multi-step14  - long-horizon15  - image-to-video16  - evaluation17  - benchmark18---19 20# VBVR-MultiStep-Bench21 22The frozen **180-instance public evaluation split** released alongside the [VBVR-MultiStep](https://huggingface.co/datasets/Video-Reason/VBVR-MultiStep) training corpus. Designed for long-horizon multi-step image-to-video (I2V) reasoning evaluation.23 24This dataset is part of the **VBVR (Very Big Video Reasoning Suite)** project. See the parent suite at <https://video-reason.com> and the suite paper [VBVR: A Very Big Video Reasoning Suite (Wang et al., ICML 2026)](https://icml.cc/virtual/2026/poster/65709).25 26## At a glance27 28| Property | Value |29|---|---|30| Tasks | **36** parameterized tasks (`Multi-01` … `Multi-36`) |31| Reasoning families | Navigation, Planning, CSP, Execution, Geometry, Physics |32| Instances | **180** (5 per task × 36) |33| Per-instance artifacts | 5 (see below) |34| License | CC-BY-4.0 |35 36## Five-artifact data contract37 38Every instance lives at:39 40```41Multi-XX_<name>_data-generator/Multi-XX_<name>_data-generator_task/Multi-XX_<name>_data-generator_<id>/42```43 44and contains exactly:45 46| File | Role |47|---|---|48| `first_frame.png` | Model conditioning image (the only visual input the model receives at inference) |49| `prompt.txt` | Natural-language task contract |50| `final_frame.png` | Target endpoint (held out from the model) |51| `ground_truth.mp4` | Reference rollout demonstrating the correct trajectory |52| `question_metadata.json` | Seed, version, tolerances, task-specific fields |53 54A top-level `metadata.parquet` indexes every instance with the task id, family, seed, and per-instance metadata for fast filtering.55 56## Reasoning families57 58| Family | Characteristic | Released tasks |59|---|---|---|60| Navigation | Discrete motion under adjacency / obstacle constraints | 6 |61| Planning | Operator-based state transformation | 6 |62| CSP | Incremental labeling under global consistency | 6 (3 used for human judging) |63| Execution | Clocked deterministic update rules | 6 |64| Geometry | Ordered constructive geometry | 6 |65| Physics | Continuous dynamics with contact / conservation | 6 |66 67Tasks `Multi-13`, `Multi-14`, `Multi-15` (CSP) are excluded from the human-judging pool described in the paper but are included in this release for completeness.68 69## Intended use70 71- **Primary use**: trajectory-level evaluation of I2V systems under a fixed five-artifact contract.72- **Comparison protocol**: blind human pairwise judging on three independent axes — process correctness, reference fidelity, render quality.73- **Companion training corpus**: [Video-Reason/VBVR-MultiStep](https://huggingface.co/datasets/Video-Reason/VBVR-MultiStep) (~360k samples).74 75## Loading76 77```python78import pandas as pd79meta = pd.read_parquet("hf://datasets/Video-Reason/VBVR-MultiStep-Bench/metadata.parquet")80```81 82Or pull a single instance:83 84```python85from huggingface_hub import hf_hub_download86prompt_path = hf_hub_download(87    "Video-Reason/VBVR-MultiStep-Bench",88    "Multi-01_maze_shortest_path_data-generator/Multi-01_maze_shortest_path_data-generator_task/Multi-01_maze_shortest_path_data-generator_00000000/prompt.txt",89    repo_type="dataset",90)91```92 93## License94 95Released under **CC-BY-4.0**. The reference rollouts are produced from generators that consume only released task definitions; no third-party copyrighted content is embedded.96 97Wan2.2-I2V-A14B (Apache-2.0) is referenced as a baseline model and a fine-tuning ancestor for `VBVR-Wan2.2`; this dataset does not redistribute Wan2.2 weights.98 99## Responsible AI100 101This dataset is fully synthetic — generators produce every instance from controlled parameters. There are no human subjects, no scraped media, and no personal information. See the [Croissant file](./croissant.json) for the complete RAI metadata.102