Fangjun/RoomSpace

<div align="left"> <h1 style="font-size: 2em;">🏠 RoomSpace Benchmark</h1> 🔗 <a href="https://github.com/Fangjun-Li/RoomSpace">GitHub</a> • 📁 <a href="https://archive.researchdata.leeds.ac.uk/1293/">Dataset</a> • 📖 <a href="https://arxiv.org/abs/2405.15064">Paper</a> 🕹️ <a href="https://colab.research.google.com/drive/1fAK8J1UHAjMm-mNVsuzIbEZd-SZG6bX-?usp=sharing">Colab</a> </div> <br/>

Branch Structure

This repository contains various branches, each representing different versions and sizes of the RoomSpace dataset. For example, the SD-100_v1.0 branch includes 100 test examples under version 1.0.

Data/
├── main/
├── SD-100_v1.0/
├── SD-1K_v1.0/
└── SD-10K_v1.0/

• —main: for the Dataset Viewer, contains the n5_m4_d144 directory for SD-1K_v1.0.
• —SD-100_v1.0: Contains data for 100 room samples.
• —SD-1K_v1.0: Contains data for 1,000 room samples, with the first 100 rooms identical to those in SD-100_v1.0.
• —SD-10K_v1.0: Contains data for 10,000 room samples, with the first 1,000 rooms identical to those in SD-1K_v1.0.

Directory Structure

The repository is organized as follows:

Revision/
├── n3_m2_d144/
├── n4_m3_d144/

Each n*_m*_d* subdirectory includes a test parquet file containing stories, questions, and answers derived from the images and a Constraint Satisfaction Problem (CSP) reasoner. The parameters n, m, and d represent the object number, constraint density, and domain size, respectively.

Images

Each example includes two images showing different views of the rooms:

• —image_top_down: Top-down images of the rooms.
• —image_north_facing: North-facing rectangular images, assuming a robot stands at the south door, facing north.

Questions & Answers

There are a total of 26 QA pairs, categorized as follows:

• —Visual QA: (imagetopdown + questionimage) -- answerimagetd, (imagenorthfacing + questionimage) -- answerimagesd
• —Layout Setting: (questiontdlayoutyn -- answertdlayoutyn), (questiontdlayoutfr -- answertdlayoutfr)
• —TPP Setting: (questiontdtppyn -- answertdtppyn), (questiontdtppfr -- answertdtppfr)
• —O2 Setting: (questiontdo2yn -- answertdo2yn), (questiontdo2fr -- answertdo2fr), (questionsdo2yn -- answersdo2yn), (questionsdo2fr -- answersdo2fr)
• —O2+D2 Setting: (questiontdo2d2yn -- answertdo2d2yn), (questiontdo2d2fr -- answertdo2d2fr), (questionsdo2d2yn -- answersdo2d2yn), (questionsdo2d2fr -- answersdo2d2fr)
• —O2+D3 Setting: (questiontdo2d3yn -- answertdo2d3yn), (questiontdo2d3fr -- answertdo2d3fr), (questionsdo2d3yn -- answersdo2d3yn), (questionsdo2d3fr -- answersdo2d3fr)
• —Layout + O2 + D2 Setting: (questiontdo2d2layoutyn -- answertdo2d2layoutyn), (questiontdo2d2layoutfr -- answertdo2d2layoutfr), (questionsdo2d2layoutyn -- answersdo2d2layoutyn), (questionsdo2d2layoutfr -- answersdo2d2layoutfr)
• —Layout + O2 + D3 Setting: (questiontdo2d3layoutyn -- answertdo2d3layoutyn), (questiontdo2d3layoutfr -- answertdo2d3layoutfr), (questionsdo2d3layoutyn -- answersdo2d3layoutyn), (questionsdo2d3layoutfr -- answersdo2d3layoutfr)

Question Types

• —'fr': Find Relation
• —'yn': Yes/No

Points of View

• —'td': Top-Down View
• —'sd': North Facing View (from the south door)

Relevant Paper

Fangjun Li, David Hogg, and Anthony Cohn. "Reframing Spatial Reasoning Evaluation in Language Models: A Real-World Simulation Benchmark for Qualitative Reasoning." IJCAI, 2024.