Andrew613/PAWBench

PAWBench

A benchmark for distributional physical realism in generated videos

PAWBench asks whether a video model reproduces the range and frequency of outcomes that a physical scene can produce. It evaluates repeated rollouts from the same source image and action, rather than treating one plausible-looking video as sufficient evidence of physical realism.

Paper · Code · Project website

At a glance

Repository layout

.
├── manifest.json
├── scenes.jsonl
├── source_images/
│   └── <scene source image>.png|jpg
└── prompts/
    └── gt_guided/
        ├── prompt_bank.jsonl
        ├── README.md
        └── review.md

manifest.json declares the package schema and points consumers to the scene table. scenes.jsonl is the benchmark contract: each line is one scene and paths are relative to this repository root.

Scene format

Every scene row contains the model input needed to generate a rollout and the outcome contract needed to evaluate it.

Illustrative Calibration record:

{
  "scene_id": "A-01",
  "split": "calibration",
  "source_image_path": "source_images/A01.png",
  "action": "Flick the coin once.",
  "outcome_labels": ["heads", "tails"],
  "reference_distribution": {"heads": 0.5, "tails": 0.5}
}

Using the data

Download this dataset, generate the complete rollout grid described by the PAWBench evaluator, and run the repository's command-line workflow:

hf download Andrew613/PAWBench \
  --repo-type dataset \
  --local-dir /path/to/PAWBench-data

git clone https://github.com/Andrew0613/PAWBench.git
cd PAWBench
pip install -r requirements.txt

python evaluate.py \
  --benchmark /path/to/PAWBench-data \
  --videos /path/to/my-model-rollouts \
  --output runs/my-model/evaluation \
  --model my-model \
  --vlm-base-url https://openrouter.ai/api/v1 \
  --vlm-model google/gemini-3.5-flash \
  --vlm-api-key-env OPENROUTER_API_KEY

The evaluator derives the full 50-scene × 50-rollout grid from the scene table. Missing or malformed items remain visible as blockers and do not shrink the benchmark denominator.

Scope

This dataset repository contains benchmark inputs only:

• —scene definitions, source images, and official prompt material are included;
• —PAWEval code and its rubric implementation live in the GitHub repository;
• —generated model videos, provider responses, and experiment result bundles are intentionally excluded.

This public repository preserves the materialized 50-scene benchmark package. Record the exact Hugging Face revision together with the evaluator commit for reproducible evaluations.

Citation

If you use PAWBench, please cite the paper and record the exact Hugging Face revision and GitHub commit used for your evaluation:

@article{pu2026pawbench,
  title={PAWBench: How Far Are We from Probabilistically Aligned World Modeling?},
  author={Yuandong Pu and Le Zhuo and Sayak Paul and Gabriel Jorge Menezes and Avram Đorđević and Shiyang Li and Yifan Zhou and Bin Fu and Wenlong Zhang and Junjun He and Yu Qiao and Yihao Liu and Jinbo Xing and Xi Chen},
  journal={arXiv preprint arXiv:2608.27345},
  year={2026},
  eprint={2608.27345},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2608.27345}
}

License and usage

The Apache-2.0 license in the PAWBench code repository applies to the evaluator source code, not automatically to this dataset's images, prompt material, or scene metadata. No separate license has been selected for this dataset repository. Do not infer permission to redistribute or create derivatives of its assets without permission from the project owners. Source and package provenance is recorded in scenes.jsonl and manifest.json.