owl-owl/POVBench
POVBench Contextual Observer Grounding: Evaluating Situated Spatial Reasoning in Vision-Language ModelsEMNLP 2026 Findings Project page · Code Given a sentence in which an observer says where they last saw an object — from their own point of view — a model must recover that perspective and localize the target in image space. Crucially, the observer's right is not necessarily aligned with the camera's right. Three conditions progressively reduce the amount of reasoning required:… See the full description on the dataset page: https://huggingface.co/datasets/owl-owl/POVBench.
POVBench
Contextual Observer Grounding: Evaluating Situated Spatial Reasoning in Vision-Language Models EMNLP 2026 Findings
Given a sentence in which an observer says where they last saw an object — from their own point of view — a model must recover that perspective and localize the target in image space.
Crucially, the observer's right is not necessarily aligned with the camera's right.
Three conditions progressively reduce the amount of reasoning required:
The steps are operational reasoning, observer grounding, and relational reasoning.
See the paper for the task definition, construction pipeline, and results. The real-world evaluation data used in the paper (Tables 5–6) is not included in this release; refer to the paper for details.
Quick start
from huggingface_hub import snapshot_download
path = snapshot_download(repo_id="owl-owl/POVBench", repo_type="dataset")Pass that path to the evaluation code as --dataset-dir.
This is a file-based dataset with a nested per-house layout, so datasets.load_dataset() and the Hub's dataset viewer do not apply. Download the folder and read the JSON directly.Layout
POVBench/
├── meta.json # statistics and provenance
├── LICENSE, NOTICE # licensing and upstream attribution
├── assets/overview.png # the figure used by this card
└── artifacts/
└── train_house_000NN/ # one directory per house, 47 in total
├── train_house_000NN.json # ← the benchmark instances
├── exploration_images/*.png # walkthrough frames
├── img_from_observer/*.png # observer-viewpoint image (condition C)
├── exploration_log.json # camera pose + visible objects per frame
├── per_image_visible_objects.json
├── gt_images/*.png # ground-truth visualizations
├── top_down_view.png # floor-plan overview
└── stage1_candidates.json, stage2..6_pairs.json # generation recordsSchema
Each scenario in train_house_000NN.json pairs an observer context (represented by the furniture associated with the speaker's activity) with an anchor (the landmark relative to which the target is described).
dir is one of front, up, left, right; a scenario holds up to four.
Enumerate instances from `stage5` / `stage6`. placements also contains directions that were filtered out of the benchmark.Inputs per condition
Ground truth
The scored ground truth is not stored in the dataset. eval_results.py in the code repository recomputes it from the anchor geometry and the stage3 surface verdict, then projects it into the image.
Contribution
Original contributions of POVBench: the task formulation and three-condition design, the capture procedure, observer–anchor pairing and target selection, ground-truth generation and filtering, and all natural-language sentences and annotations.
Built on ProcTHOR / AI2-THOR
The 3D environments are not ours. The houses were procedurally generated with ProcTHOR (Deitke et al., 2022), built on AI2-THOR (Kolve et al., 2017).
Both are from the Allen Institute for AI under the Apache License 2.0. Please cite them alongside this work. Required attribution is in NOTICE.
License
See LICENSE and NOTICE for the full terms, and the paper's Limitations section for known caveats.
Citation
@inproceedings{shirasaka2026povbench,
title = {Contextual Observer Grounding: Evaluating Situated Spatial
Reasoning in Vision-Language Models},
author = {Shirasaka, Mimo and Zhang, Haochen and Bisk, Yonatan},
booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
year = {2026}
}