manycore-research/SpatialLM-Dataset

1---2license: cc-by-nc-4.03configs:4  - config_name: default5    data_files: "split.csv"6---7 8# SpatialLM Dataset9 10<!-- markdownlint-disable first-line-h1 -->11<!-- markdownlint-disable html -->12<!-- markdownlint-disable no-duplicate-header -->13 14<div align="center">15  <picture>16    <source srcset="https://cdn-uploads.huggingface.co/production/uploads/63efbb1efc92a63ac81126d0/_dK14CT3do8rBG3QrHUjN.png" media="(prefers-color-scheme: dark)">17    <img src="https://cdn-uploads.huggingface.co/production/uploads/63efbb1efc92a63ac81126d0/bAZyeIXOMVASHR6-xVlQU.png" width="60%" alt="SpatialLM""/>18  </picture>19</div>20<hr style="margin-top: 0; margin-bottom: 8px;">21<div align="center" style="margin-top: 0; padding-top: 0; line-height: 1;">22    <a href="https://manycore-research.github.io/SpatialLM" target="_blank" style="margin: 2px;"><img alt="Project"23    src="https://img.shields.io/badge/🌐%20Website-SpatialLM-ffc107?color=42a5f5&logoColor=white" style="display: inline-block; vertical-align: middle;"/></a>24    <a href="https://arxiv.org/abs/2506.07491" target="_blank" style="margin: 2px;"><img alt="arXiv"25    src="https://img.shields.io/badge/arXiv-Techreport-b31b1b?logo=arxiv&logoColor=white" style="display: inline-block; vertical-align: middle;"/></a>26    <a href="https://github.com/manycore-research/SpatialLM" target="_blank" style="margin: 2px;"><img alt="GitHub"27    src="https://img.shields.io/badge/GitHub-SpatialLM-24292e?logo=github&logoColor=white" style="display: inline-block; vertical-align: middle;"/></a>28</div>29<div align="center" style="line-height: 1;">30    <a href="https://huggingface.co/manycore-research/SpatialLM1.1-Qwen-0.5B" target="_blank" style="margin: 2px;"><img alt="Hugging Face"31    src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-SpatialLM-ffc107?color=ffc107&logoColor=white" style="display: inline-block; vertical-align: middle;"/></a>32    <a href="https://huggingface.co/datasets/manycore-research/SpatialLM-Dataset" target="_blank" style="margin: 2px;"><img alt="Dataset"33    src="https://img.shields.io/badge/%F0%9F%A4%97%20Dataset-Dataset-ffc107?color=ffc107&logoColor=white" style="display: inline-block; vertical-align: middle;"/></a>34    <a href="https://huggingface.co/datasets/manycore-research/SpatialLM-Testset" target="_blank" style="margin: 2px;"><img alt="Dataset"35    src="https://img.shields.io/badge/%F0%9F%A4%97%20Dataset-Testset-ffc107?color=ffc107&logoColor=white" style="display: inline-block; vertical-align: middle;"/></a>36</div>37 38The SpatialLM dataset is a large-scale, high-quality synthetic dataset designed by professional 3D designers and used for real-world production. It contains point clouds from 12,328 diverse indoor scenes comprising 54,778 rooms, each paired with rich ground-truth 3D annotations. SpatialLM dataset provides an additional valuable resource for advancing research in indoor scene understanding, 3D perception, and related applications. For more details about the dataset construction, annotations, and benchmark tasks, please refer to the [paper](https://arxiv.org/abs/2506.07491).39 40<table style="table-layout: fixed;">41  <tr>42    <td style="text-align: center; vertical-align: middle; width: 25%"> <img src="https://cdn-uploads.huggingface.co/production/uploads/63efbb1efc92a63ac81126d0/YFQzBUC_sGufXqpGL6YhV.jpeg"  alt="exmaple a" width="100%" style="display: block;"></td>43    <td style="text-align: center; vertical-align: middle; width: 25%"> <img src="https://cdn-uploads.huggingface.co/production/uploads/63efbb1efc92a63ac81126d0/jRbPzBwhtDMWUwueodYax.jpeg" alt="exmaple c" width="100%" style="display: block;"></td>44    <td style="text-align: center; vertical-align: middle; width: 25%"> <img src="https://cdn-uploads.huggingface.co/production/uploads/63efbb1efc92a63ac81126d0/DpNKunoD-2-1spx6cXDxa.jpeg" alt="exmaple b" width="100%" style="display: block;"></td>45    <td style="text-align: center; vertical-align: middle; width: 25%"> <img src="https://cdn-uploads.huggingface.co/production/uploads/63efbb1efc92a63ac81126d0/o-JgD-oY0oK0yhryWUexv.jpeg" alt="exmaple d" width="100%" style="display: block;"></td>46   </tr> 47  </tr>48</table>49 50## Dataset Structure51 52The dataset is organized into the following folder structure:53 54```bash55SpatialLM-Dataset/56├── pcd/                        # Point cloud PLY files for rooms57│ └── .ply58├── layout/                     # GT room layout59│ └── .txt60├── examples/                   # 10 point cloud and layout examples61│ └── .ply62│ └── .txt63├── extract.sh                  # Extraction script64├── dataset_info.json           # Dataset configuration file for training65├── spatiallm_train.json        # SpatialLM conversations data for training66├── spatiallm_val.json          # SpatialLM conversations data for validation67├── spatiallm_test.json         # SpatialLM conversations data for testing68└── split.csv                   # Metadata CSV file69```70 71## Metadata72 73The dataset metadata is provided in the `split.csv` file with the following columns:74 75- **id**: Unique identifier for each sampled point cloud and layout following the naming convention `{scene_id}_{room_id}_{sample}` (e.g., `scene_001523_00_2`)76- **room_type**: The functional type of each room (e.g., bedroom, living room)77- **scene_id**: Unique identifier for multi-room apartment scenes78- **room_id**: Unique identifier for individual rooms within a scene79- **sample**: Point cloud sampling configuration for each room (4 types available):80  - **0**: Most complete observations (8 panoramic views randomly sampled)81  - **1**: Most sparse observations (8 perspective views randomly sampled)82  - **2**: Less complete observations (16 perspective views randomly sampled)83  - **3**: Less sparse observations (24 perspective views randomly sampled)84- **split**: Dataset partition assignment (`train`, `val`, `test`, `reserved`)85 86The dataset is divided into 11,328/500/500 scenes for train/val/test splits, and 199,286/500/500 sampled point clouds accordingly, where multiple point cloud samples of the same room are randomly selected for the val/test splits for simplicity.87 88## Data Extraction89 90Point clouds and layouts are compressed in zip files. To extract the files, run the following script:91 92```bash93cd SpatialLM-Dataset94chmod +x extract.sh95./extract.sh96```97 98## Conversation Format99 100The `spatiallm_train.json`, `spatiallm_val.json`, and `spatiallm_test.json` data follows the **SpatialLM format** with ShareGPT-style conversations:101 102```json103{104  "conversations": [105    {106      "from": "human",107      "value": "<point_cloud>Detect walls, doors, windows, boxes. The reference code is as followed: ..."108    },109    {110      "from": "gpt",111      "value": "<|layout_s|>wall_0=...<|layout_e|>"112    }113  ],114  "point_clouds": ["pcd/ID.ply"]115}116```117 118## Usage119 120Use the [SpatialLM code base](https://github.com/manycore-research/SpatialLM/tree/main) for reading the point cloud and the layout data.121 122```python123from spatiallm import Layout124from spatiallm.pcd import load_o3d_pcd125 126# Load Point Cloud127point_cloud = load_o3d_pcd(args.point_cloud)128 129# Load Layout130with open(args.layout, "r") as f:131    layout_content = f.read()132layout = Layout(layout_content)133```134 135## Visualization136 137Use `rerun` to visualize the point cloud and the GT structured 3D layout output:138 139```bash140python visualize.py --point_cloud examples/scene_008456_00_3.ply --layout examples/scene_008456_00_3.txt --save scene_008456_00_3.rrd141rerun scene_008456_00_3.rrd142```143 144## SpatialGen dataset145 146For access to photorealistic RGB/Depth/Normal/Semantic/Instance panoramic renderings and camera trajectories used to generate the SpatialLM point clouds, please refer to the [SpatialGen project](https://manycore-research.github.io/SpatialGen) for more details.147 148## Citation149 150If you find this work useful, please consider citing:151 152```bibtex153@inproceedings{SpatialLM,154  title     = {SpatialLM: Training Large Language Models for Structured Indoor Modeling},155  author    = {Mao, Yongsen and Zhong, Junhao and Fang, Chuan and Zheng, Jia and Tang, Rui and Zhu, Hao and Tan, Ping and Zhou, Zihan},156  booktitle = {Advances in Neural Information Processing Systems},157  year      = {2025}158}159```160