Andrew613/PAWBench

1---2size_categories:3- n<1K4pretty_name: PAWBench5task_categories:6- image-to-video7tags:8- benchmark9- video-generation10- physical-reasoning11---12 13# PAWBench14 15### A benchmark for distributional physical realism in generated videos16 17PAWBench asks whether a video model reproduces the *range and frequency* of18outcomes that a physical scene can produce. It evaluates repeated rollouts from19the same source image and action, rather than treating one plausible-looking20video as sufficient evidence of physical realism.21 22[Paper](https://arxiv.org/abs/2608.27345) ·23[Code](https://github.com/Andrew0613/PAWBench) ·24[Project website](https://pawbench.github.io/)25 26## At a glance27 28| Component | Contents |29| --- | --- |30| Scenes | 50 physical scenes |31| Calibration track | 25 scenes with reference outcome distributions |32| Coverage track | 25 scenes with supported outcome labels |33| Source images | 50 first-frame images |34| Prompt material | One base prompt per scene and the `gt_guided` prompt bank |35| Evaluator | [PAWBench code](https://github.com/Andrew0613/PAWBench) |36 37## Repository layout38 39```text40.41├── manifest.json42├── scenes.jsonl43├── source_images/44│   └── <scene source image>.png|jpg45└── prompts/46    └── gt_guided/47        ├── prompt_bank.jsonl48        ├── README.md49        └── review.md50```51 52`manifest.json` declares the package schema and points consumers to the scene53table. `scenes.jsonl` is the benchmark contract: each line is one scene and54paths are relative to this repository root.55 56## Scene format57 58Every scene row contains the model input needed to generate a rollout and the59outcome contract needed to evaluate it.60 61| Field | Meaning |62| --- | --- |63| `scene_id` | Stable scene identifier |64| `split` | `calibration` or `coverage` |65| `source_image_path` | Local first-frame image path |66| `action` | Physical action to be performed |67| `base_prompt` | Baseline image-to-video prompt |68| `outcome_labels` | Canonical labels for observable outcomes |69| `reference_distribution` | Calibration-only target distribution; `null` for Coverage |70 71Illustrative Calibration record:72 73```json74{75  "scene_id": "A-01",76  "split": "calibration",77  "source_image_path": "source_images/A01.png",78  "action": "Flick the coin once.",79  "outcome_labels": ["heads", "tails"],80  "reference_distribution": {"heads": 0.5, "tails": 0.5}81}82```83 84## Using the data85 86Download this dataset, generate the complete rollout grid described by the87[PAWBench evaluator](https://github.com/Andrew0613/PAWBench), and run the88repository's command-line workflow:89 90```bash91hf download Andrew613/PAWBench \92  --repo-type dataset \93  --local-dir /path/to/PAWBench-data94 95git clone https://github.com/Andrew0613/PAWBench.git96cd PAWBench97pip install -r requirements.txt98 99python evaluate.py \100  --benchmark /path/to/PAWBench-data \101  --videos /path/to/my-model-rollouts \102  --output runs/my-model/evaluation \103  --model my-model \104  --vlm-base-url https://openrouter.ai/api/v1 \105  --vlm-model google/gemini-3.5-flash \106  --vlm-api-key-env OPENROUTER_API_KEY107```108 109The evaluator derives the full 50-scene × 50-rollout grid from the scene110table. Missing or malformed items remain visible as blockers and do not shrink111the benchmark denominator.112 113## Scope114 115This dataset repository contains benchmark inputs only:116 117- scene definitions, source images, and official prompt material are included;118- PAWEval code and its rubric implementation live in the GitHub repository;119- generated model videos, provider responses, and experiment result bundles are120  intentionally excluded.121 122This public repository preserves the materialized 50-scene benchmark package.123Record the exact Hugging Face revision together with the evaluator commit for124reproducible evaluations.125 126## Citation127 128If you use PAWBench, please cite the paper and record the exact Hugging Face129revision and GitHub commit used for your evaluation:130 131```bibtex132@article{pu2026pawbench,133  title={PAWBench: How Far Are We from Probabilistically Aligned World Modeling?},134  author={Yuandong Pu and Le Zhuo and Sayak Paul and Gabriel Jorge Menezes and Avram Đorđević and Shiyang Li and Yifan Zhou and Bin Fu and Wenlong Zhang and Junjun He and Yu Qiao and Yihao Liu and Jinbo Xing and Xi Chen},135  journal={arXiv preprint arXiv:2608.27345},136  year={2026},137  eprint={2608.27345},138  archivePrefix={arXiv},139  primaryClass={cs.CV},140  url={https://arxiv.org/abs/2608.27345}141}142```143 144## License and usage145 146The Apache-2.0 license in the147[PAWBench code repository](https://github.com/Andrew0613/PAWBench) applies to148the evaluator source code, not automatically to this dataset's images, prompt149material, or scene metadata. No separate license has been selected for this150dataset repository. Do not infer permission to redistribute or create151derivatives of its assets without permission from the project owners. Source152and package provenance is recorded in `scenes.jsonl` and `manifest.json`.