Datasetpublic

leopoldmaillard/sceneteract-grpo

SceneTeract GRPO Training Set Action-level feasibility samples for post-training a VLM against a geometric verifier. Each row is one atomic interaction — an image, a prompt, and a label that was measured rather than annotated — ready to drop into TRL's GRPOTrainer. 8,073 samples over 1,132 3D-FRONT living rooms and dining rooms and three agent profiles. from datasets import load_dataset ds = load_dataset("leopoldmaillard/sceneteract-grpo") ds["train"] # 6,473 samples / 905… See the full description on the dataset page: https://huggingface.co/datasets/leopoldmaillard/sceneteract-grpo.

sourceHugging Facecc-by-nc-4.0updated 8d agoView on Hugging Face

0likes96downloads

Dataset Card

SceneTeract GRPO Training Set

Action-level feasibility samples for post-training a VLM against a geometric verifier. Each row is one atomic interaction — an image, a prompt, and a label that was measured rather than annotated — ready to drop into TRL's GRPOTrainer.

8,073 samples over 1,132 3D-FRONT living rooms and dining rooms and three agent profiles.

python

from datasets import load_dataset

ds = load_dataset("leopoldmaillard/sceneteract-grpo")
ds["train"]   # 6,473 samples /  905 scenes
ds["test"]    # 1,600 samples /  227 scenes

Columns

column	type
`image`	Image	top-down view of the room with the target object in a red bounding box
`messages`	list⟨struct⟩	system + user turns, with an image placeholder, in TRL's chat format
`ground_truth`	string	`"True"` / `"False"` — a string, not a bool, as TRL reward functions compare strings
`failure_property`	string	the check that failed, or the string `"None"` when the action is feasible
`action`	string	the atomic action, e.g. `open`, `sit_on`, `take_out_of`
`agent_id`	string	`adult`, `child`, `adult_wheelchair`
`scene_id`	string	3D-FRONT scene, as preprocessed by PhyScene

Both ground_truth and failure_property are strings including their "empty" cases — "False" and "None" are values, not nulls. Compare them as strings.

Splits

Scene-disjoint, 80/20. The same partition ships in `sceneteract-traces`, so a model trained here can be evaluated on the traces' test split without leakage.

Training

ground_truth is the reward signal: a rollout is correct when its predicted feasibility matches the verifier's measurement. The reward functions used — format, correctness, and spatial shaping — ship with the code:

bash

pip install "sceneteract[grpo]"
sceneteract-grpo train --dataset <path> --model Qwen/Qwen3-VL-4B-Instruct

See docs/usecase_c_grpo.md for reward shaping and hyperparameters.

Licence and attribution

Released under *CC BY-NC 4.0.

If you use this dataset, please cite those works alongside SceneTeract:

—3D-FRONT — Fu et al., 3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics, ICCV 2021
—3D-FUTURE — Fu et al., 3D-FUTURE: 3D Furniture Shape with TextURE, IJCV 2021
—PhyScene — Yang et al., PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI, CVPR 2024

Citation

bibtex

@misc{maillard2026sceneteract,
      title={{SceneTeract: Probing and Improving Agent-Aware Activity Reasoning in 3D Indoor Scenes}}, 
      author={Léopold Maillard and Francis Engelmann and Tom Durand and Boxiao Pan and Yang You and Leonidas Guibas and Maks Ovsjanikov},
      year={2026},
      eprint={2603.29798},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2603.29798}, 
}