WorldModelBenchmark/playworld-bench-anonymous

PlayWorld: Benchmarking World Models with Agent Players over Long-Horizon Objectives

Anonymous benchmark release for double-blind review. Author, affiliation, and identity-bearing links are intentionally omitted.

**Project** | **Code**

Video world models generate future states from an initial observation and user actions. Comparing interactive models fairly is difficult because the actions required to reach the same long-horizon objective can vary substantially across models.

PlayWorld evaluates world models from the perspective of a human player. Each case provides an initial world, a scene-grounded long-horizon objective, a human-annotated basic action sequence, and a sample-specific VQA rubric. During evaluation, a multi-modal Agent Player observes generated frames and adapts action execution toward the objective. This dataset provides the initial worlds, objectives, basic action sequences, and rubrics used by that protocol.

Benchmark Overview

PlayWorld contains 171 human-annotated cases and 820+ sample-specific VQA questions across four core world-model capabilities:

• —Geometry Consistency (GC): whether object identity, appearance, relative position, and scene layout remain consistent during camera movement and revisitation.
• —Interaction Fidelity (IF): whether a controlled subject reaches and interacts with the intended region through plausible contact, collision, motion, causal response, and visual feedback.
• —Insight Evolution (IE): whether a continuously observed subject or process exhibits coherent state progression, physically plausible motion, and temporally consistent surroundings.
• —Out-of-sight Evolution (OE): whether a target preserves its identity and continues to evolve plausibly while outside the field of view, with a causally consistent state when revealed again.

The questions cover four broad evidence families: identity and appearance, physics and dynamics, temporal evolution, and spatial and trajectory reasoning. Rubrics are tailored to each scene so that only observable and applicable criteria are evaluated.

Dataset Composition

Insight and out-of-sight cases retain their original OExxx task IDs and image filenames. IDs are not renumbered after the two evaluation settings are placed in separate configs.

Initial worlds cover natural, urban, indoor, and imaginative environments, with human, animal, vehicle, object, and scene-level subjects. Images were selected for clear, visually verifiable structures and use first- or third-person views according to the target capability.

Data Format

datasuite/
├── README.md
├── gc/
│   ├── data.json
│   └── images/
├── if/
│   ├── data.json
│   └── images/
├── insight/
│   ├── data.json
│   └── images/
└── outsight/
    ├── data.json
    └── images/

Each data.json is an array of evaluation cases. A case contains:

• —task_id, source_task_id, category, and suite_split;
• —image_path, image_caption, and perspective for the initial world;
• —prompt for the scene-grounded long-horizon objective;
• —action, action_sequence, and action_sequence_steps for the annotated basic action reference;
• —questions, including the evaluation category, weight, question text, and scoring notes;
• —dimension- or case-specific evaluation policies where applicable.

Download

This folder is the complete local benchmark used by the companion code release. To obtain another copy from a future anonymous dataset host, download it into the code repository's data/ directory:

hf download <anonymous-dataset-repository> \
  --repo-type dataset \
  --local-dir data

If access is restricted, request access on the dataset page and authenticate first with hf auth login.

Individual configs can also be loaded with datasets:

from datasets import load_dataset

gc = load_dataset("<anonymous-dataset-repository>", "gc", split="train")
interaction = load_dataset("<anonymous-dataset-repository>", "if", split="train")
insight = load_dataset("<anonymous-dataset-repository>", "insight", split="train")
outsight = load_dataset("<anonymous-dataset-repository>", "outsight", split="train")

Download the complete repository when running PlayWorld so that every relative image_path resolves from the same dataset root.

Use with PlayWorld

Keep the code and dataset as separate directories:

workspace/
├── PlayWorld/       # code repository
└── data/         # this dataset

Install the evaluation code from the sibling `../code/` directory, then point the Agent Player and Rubric Verifier to the selected data.json and its corresponding initial image. For example, the VQA evaluation entry point uses:

GEMINI_API_KEY="xxx" \
DATASET="$PWD/data/gc/data.json" \
TASK_ID=GC002 \
VIDEO="$PWD/outputs/genie3/GC002/GC002_full_process.mp4" \
REFERENCE_IMAGE="$PWD/data/gc/images/GC002.jpg" \
OUTPUT="$PWD/evaluation/genie3/GC002_score.json" \
./run_vqa_score.sh

See the code repository for Agent Player integration, world-model adapters, and the complete inference and evaluation protocol.

License

Initial images originate from multiple sources, including Pexels and web image search results. Dataset users must follow the source-specific license and redistribution restrictions applicable to each asset. A single permissive license must not be assumed to cover every image.

Citation

For double-blind review, cite the accompanying anonymous manuscript:

@article{playworld_anonymous,
  title  = {PlayWorld: Benchmarking World Models with Agent Players over Long-Horizon Objectives},
  author = {Anonymous Authors},
  year   = {2026}
}