VideoWorldmodel/Evaluation

Reasoning-Structured Videos: Evaluation Artifacts

This page hosts evaluation artifacts for Reasoning-Structured Videos: A Stratified Diagnostic Suite for Compositional Consistency in World Models.

The benchmark tests whether action-conditioned video world models respect three trajectory relations:

• —Inverse: a path followed by its inverse should return to the initial state.
• —Loop: a closed path should return to the initial state.
• —Equivalence: two different paths reaching the same state should produce matching endpoints.

We report self-consistency (SC) as the primary cross-model diagnostic and use GT-anchor and distributional metrics as complementary evidence.

Files

Reference videos and trajectory metadata are hosted separately in VideoWorldmodel/ReasoningStructureTestset.

Released R50 rollouts

The r50/ directory contains the generated rollouts used for the frozen R50 evaluation of two camera-trajectory-conditioned methods:

R50 has 50 graphs for each of Inverse, Loop, and Equivalence. Each tier contains 200 rollouts because every Equivalence graph has paired A/B branches. Self-consistency rollouts start from the model's conditioning frame; GT-anchor rollouts condition on the first 180 GT frames and predict through logical raw359.

r50/
  manifests/subset_manifest_R20_R50_raw360_v1.jsonl
  minwm/{self_consistency,gt_anchor}/predictions/<relation>/<record_id>/
  hy_worldplay/{self_consistency,gt_anchor}/predictions/<relation>/<record_id>/

Each record includes rollout.mp4 and metadata.json; reference.png is included where produced by the inference adapter. Use the logical-to-physical frame mapping in metadata.json when selecting frames: saved video spans differ between SC and GT-anchor outputs, so physical frame indices must not be guessed.

Download only this release with:

hf download VideoWorldmodel/Evaluation \
  --repo-type dataset \
  --include "r50/**" \
  --local-dir .

Main results

Common camera-trajectory SC (frozen R50)

minWM and HY-WorldPlay receive the same frozen graph IDs and camera trajectories through their official pose-control pathways. Each model completed 200/200 rollouts with zero evaluation failures, using the same 512x288 evaluation preprocessing.

R50 contains 50 graphs per relation: 150 graph units and 200 videos because every Equivalence graph has A/B branches. The code archive includes graph-bootstrap intervals, the corresponding GT-anchor summaries, and frozen R50 Endpoint FID/KID reports. Endpoint KID uses an unbiased finite-sample estimator, so valid negative estimates are retained in the detailed artifact rather than interpreted as distances below zero.

External-model Inverse SC versus revisit horizon

On the same frozen R20 Inverse graphs, pose-certified matched-state revisits show increasing inconsistency from 10 to 40 action steps. minWM changes by +0.2697 LPIPS [0.2196, 0.3256] and -3.65 dB PSNR [-4.69, -2.80]; HY-WorldPlay changes by +0.1039 [0.0379, 0.1754] and -1.94 dB [-3.37, -0.83]. The paired GT same-pose floor shows no detectable 10-to-40 degradation. The archive contains the evaluator, tests, per-graph pose audit, strict fixed-pair sensitivity analysis, and frozen JSON/Markdown results.

Matrix-Game Inverse SC versus revisit horizon

All 448 Inverse graphs are evaluated at matched-state revisits with increasing action separation. “Physical frame” denotes the zero-indexed position in the saved Matrix MP4. Matrix exports 357 frames for the complete 40-action schedule, so action boundary k is mapped to physical frame round(356k/40), anchoring both endpoints and preserving outward/return symmetry.

The paired full-minus-10 change is +0.2211 LPIPS [0.1988, 0.2434] and -3.58 dB PSNR [-4.02, -3.15]. Adjacent boundary-rounding schemes preserve the monotonic trend and change intermediate means by at most 0.005 LPIPS / 0.10 dB.

Endpoint distribution fidelity

Endpoint FID/KID compare generated and matched GT logical raw359 endpoint sets using clean-fid Inception-v3 pool3 features.

Matrix-Game, complete available output set:

minWM, frozen R50 SC outputs:

Endpoint FID is an image-set metric and is not numerically comparable to the clip-level FVD reported elsewhere. The Matrix analysis also includes 20/50/100/200/full graph sensitivity and a GT-vs-GT finite-sample floor. These endpoint metrics complement, rather than replace, paired SC LPIPS/PSNR.

Terminal-window distribution robustness

Because an external-model R50 relation contains only 50 strict endpoint images, we additionally pool the final matched logical frames from the common SC rollout track. K=9 covers the final 0.56 seconds and K=18 the final 1.12 seconds of each 22.5-second rollout. Each entry is FID / KID ×1000.

These are point estimates over 50 graph clusters per relation. Consecutive frames are correlated, so the graph remains the sampling unit. This is a late-horizon marginal appearance-distribution check, not an exact-state closure or temporal-order metric. The code archive contains the evaluator, unit test, frozen JSON/Markdown results, and a generic reproduction command.

Quick reproduction: Matrix-Game SC

This path requires only the files on the current page; no GT dataset is needed.

python -m pip install -U huggingface_hub

hf download VideoWorldmodel/Evaluation \
  matrix_game.zip evaluate_matrix_game_sc.py \
  --repo-type dataset \
  --local-dir .

unzip matrix_game.zip -d data/MatrixGame2_SC_videos

python -m pip install \
  "numpy>=1.26,<3" \
  "opencv-python-headless>=4.8,<5" \
  "torch>=2.2,<3" \
  "torchvision>=0.17,<1" \
  "lpips==0.1.4"

python evaluate_matrix_game_sc.py \
  --data data/MatrixGame2_SC_videos \
  --output results/matrix_game_sc \
  --lpips \
  --device auto \
  --check-paper

If the archive creates one additional top-level directory, point --data to the directory that directly contains the five relation folders.

Expected values:

The evaluator verifies file counts and Equivalence pair integrity, and writes per-graph scores, graph-bootstrap confidence intervals, an audit, and a paper-value check.

Reproduce the additional analyses

Download and unpack the code artifact:

hf download VideoWorldmodel/Evaluation \
  Reasoning-Structured-Videos-Rebuttal-main.zip \
  --repo-type dataset \
  --local-dir .

unzip Reasoning-Structured-Videos-Rebuttal-main.zip
cd Reasoning-Structured-Videos-Rebuttal-main

python -m pip install -r requirements-eval.txt
python -m unittest discover -s tests -v

The repository README documents six reproducibility paths:

1. 1.recompute SC/GT LPIPS and PSNR from minWM or HY-WorldPlay rollout directories;
2. 2.recompute pose-certified Inverse SC versus horizon for minWM and HY-WorldPlay from saved rollouts;
3. 3.recompute Matrix-Game Inverse SC versus revisit horizon directly from matrix_game.zip;
4. 4.recompute relation-wise endpoint FID/KID;
5. 5.recompute R50 terminal-window FID/KID from saved external-model rollouts;
6. 6.run the full Matrix endpoint audit using matrix_game.zip and the GT dataset.

The generated minWM/HY-WorldPlay R50 rollouts are released under r50/; model checkpoints are not included. Regenerating the videos requires the official minWM or HY-WorldPlay repository, its checkpoint, its official environment, and suitable GPUs.

Frozen identifiers

• —Public subset seed: 2357
• —R20 manifest SHA-256: f8307b78ffb4633b1d6ca6340498a9db615dca4cc45a045d16139abf4375abdf
• —Nested R20/R50 manifest SHA-256: c1b3e051ec861f6babd00f67cf7c31d0ceba590d93a6fac1ac6a768b40448db4
• —matrix_game.zip SHA-256: caec3d8deb20cffbc645f0aeeaa2af425536715eef935b17347cdfab728a88d4
• —Reasoning-Structured-Videos-Rebuttal-main.zip SHA-256: 9ee07bfff222800b2eddf6d3e4a44dd2e93a250ea8e7945dbbdf1a7af8b80ca0

Scope

The released experiments evaluate deterministic camera trajectories in static scenes. They support relation-specific diagnostic conclusions under the stated protocol; they are not intended as a universal ranking of world-model architectures.