sylvainHellin/ifc-bench

IFC-Bench

A benchmark dataset for evaluating BIM (Building Information Modeling) comprehension and reasoning capabilities in AI systems. Provides curated IFC models with question-answer pairs across 4 complexity categories for testing BIM-related AI implementations.

Dataset snapshot:

Table of Contents

• —Features
• —Dataset Structure
• —Models Overview
• —Question Taxonomy
• —Getting Started
• —Dataset Integrity
• —Contributing
• —License
• —Citation
• —Acknowledgments

Features

• —22 BIM projects with 51 IFC models across architectural, structural, MEP, and specialty disciplines
• —1,027 question-answer pairs covering diverse BIM information retrieval tasks
• —4 question categories based on data retrieval complexity (direct retrieval, computational aggregation, geometric/spatial computation, incomplete information)
• —Rich contextual data: original IFC files, model cards, license documentation
• —Machine-readable format: CSV dataset with clear column structure
• —Versioned: V2 dataset with V1 preserved for backwards compatibility

Dataset Structure

ifc-bench/
├── projects/
│   ├── 4351/
│   ├── ac20/
│   ├── city_house_munich/
│   ├── dental_clinic/
│   ├── digital_hub/
│   ├── duplex/
│   ├── ettenheim_gis/
│   ├── fantasy_hotel_1/
│   ├── fantasy_hotel_2/
│   ├── fantasy_office_building_1/
│   ├── fantasy_office_building_2/
│   ├── fantasy_office_building_3/
│   ├── fantasy_residential_building_1/
│   ├── fzk_house/
│   ├── hitos/
│   ├── molio/
│   ├── samuel_macalister_sample_house/
│   ├── schependomlaan/
│   ├── sixty5/
│   ├── smiley_west/
│   ├── wbdg_office/
│   └── west_riverside_hospital/
│       ├── arc.ifc            # IFC model files
│       ├── model_card.md      # Project metadata
│       ├── license.txt        # Model license
│       └── snapshot.png       # Visual snapshot
├── questions/
│   ├── ifc-bench-v2.csv       # V2 dataset (1,027 QA pairs)
│   └── ifc-bench-v1.csv       # V1 dataset (preserved)
└── docs/
    └── CONTRIBUTING.md

Models Overview

<table> <tr> <td align="center"><img src="projects/4351/snapshot.png" width="250"/><br><b>4351</b></td> <td align="center"><img src="projects/ac20/snapshot.png" width="250"/><br><b>ac20</b></td> <td align="center"><img src="projects/cityhousemunich/snapshot.png" width="250"/><br><b>cityhousemunich</b></td> </tr> <tr> <td align="center"><img src="projects/dentalclinic/snapshot.png" width="250"/><br><b>dentalclinic</b></td> <td align="center"><img src="projects/digitalhub/snapshot.png" width="250"/><br><b>digitalhub</b></td> <td align="center"><img src="projects/duplex/snapshot.png" width="250"/><br><b>duplex</b></td> </tr> <tr> <td align="center"><img src="projects/ettenheimgis/snapshot.png" width="250"/><br><b>ettenheimgis</b></td> <td align="center"><img src="projects/fantasyhotel1/snapshot.png" width="250"/><br><b>fantasyhotel1</b></td> <td align="center"><img src="projects/fantasyhotel2/snapshot.png" width="250"/><br><b>fantasyhotel2</b></td> </tr> <tr> <td align="center"><img src="projects/fantasyofficebuilding1/snapshot.png" width="250"/><br><b>fantasyofficebuilding1</b></td> <td align="center"><img src="projects/fantasyofficebuilding2/snapshot.png" width="250"/><br><b>fantasyofficebuilding2</b></td> <td align="center"><img src="projects/fantasyofficebuilding3/snapshot.png" width="250"/><br><b>fantasyofficebuilding3</b></td> </tr> <tr> <td align="center"><img src="projects/fantasyresidentialbuilding1/snapshot.png" width="250"/><br><b>fantasyresidentialbuilding1</b></td> <td align="center"><img src="projects/fzkhouse/snapshot.png" width="250"/><br><b>fzkhouse</b></td> <td align="center"><img src="projects/hitos/snapshot.png" width="250"/><br><b>hitos</b></td> </tr> <tr> <td align="center"><img src="projects/molio/snapshot.png" width="250"/><br><b>molio</b></td> <td align="center"><img src="projects/samuelmacalistersamplehouse/snapshot.png" width="250"/><br><b>samuelmacalistersamplehouse</b></td> <td align="center"><img src="projects/schependomlaan/snapshot.png" width="250"/><br><b>schependomlaan</b></td> </tr> <tr> <td align="center"><img src="projects/sixty5/snapshot.png" width="250"/><br><b>sixty5</b></td> <td align="center"><img src="projects/smileywest/snapshot.png" width="250"/><br><b>smileywest</b></td> <td align="center"><img src="projects/wbdgoffice/snapshot.png" width="250"/><br><b>wbdgoffice</b></td> </tr> </table>

Question Taxonomy

Questions are classified into 4 categories based on data retrieval complexity and computational requirements, adapted from the classification framework by Solihin et al. (2015) for automated BIM compliance checking.

For details on how the taxonomy is used in evaluation, see:

Hellin, S., Nousias, S., & Borrmann, A. (2026). Evaluating AI Systems for BIM Information Retrieval: A Multi-Dimensional Framework Beyond Binary Accuracy. GNI 2026.

Getting Started

Prerequisites

• —Python 3.8+
• —uv (recommended for virtual environment management)

Quick Start

git clone https://github.com/sylvainHellin/ifc-bench.git
cd ifc-bench

# Install dependencies (using uv)
uv pip install pandas ifcopenshell

Using the Dataset

import pandas as pd

# Load V2 dataset
df = pd.read_csv('questions/ifc-bench-v2.csv')

print(f"Total QA pairs: {len(df)}")
print(f"Projects: {df['project'].nunique()}")
print(f"Category distribution:\n{df['category'].value_counts().sort_index()}")

# Filter by category
spatial_questions = df[df['category'] == 3]
print(f"\nGeometric/Spatial questions: {len(spatial_questions)}")

# Filter by project
duplex_questions = df[df['project'] == 'duplex']
print(f"Duplex questions: {len(duplex_questions)}")

Dataset Columns

Dataset Integrity

Verify dataset integrity using SHA-256 checksums:

# V2 dataset
shasum -a 256 questions/ifc-bench-v2.csv
# Expected: 93c293924899e6b0cd2d77cf2e7652189247582d3c85fb8104683d6b71e848d6

# V1 dataset (preserved for backwards compatibility)
shasum -a 256 questions/ifc-bench-v1.csv
# Expected: f67a48770d74b6e0ff0868c923c3e1d976110350b2c439564d7ceccc16a46f35

Contributing

We welcome contributions:

• —New IFC models (with permissive licensing)
• —Additional QA pairs for existing models
• —Documentation improvements
• —Error corrections in existing answers

Please see our Contribution Guidelines for details.

License

• —Dataset (question-answer pairs, documentation, model cards, and all original content): Licensed under CC BY 4.0.
• —IFC models: Each model retains its original license as specified in its project's license.txt. The majority of models are licensed under CC BY 4.0 or MIT. However, the following models are licensed under GNU GPLv3 and are subject to its copyleft terms:
• —4351, ettenheim_gis, hitos, samuel_macalister_sample_house (source: BIMserver TestFiles)

Users who redistribute or create derivative works that include these GPLv3 model files must comply with the GPLv3 license terms for those files. See each project's license.txt for details.

Citation

If using IFC-Bench in research, please cite the paper that introduces the v2 benchmark (currently a preprint, submitted to Automation in Construction; this entry will be updated once the paper is accepted):

@misc{hellinBIMInformationExtraction2026,
  title = {{{BIM Information Extraction Through LLM-based Adaptive Exploration}}},
  author = {Hellin, Sylvain and Jang, Suhyung and Fuchs, Stefan and Nousias, Stavros and Borrmann, Andr{\'e}},
  year = 2026,
  eprint = {2605.01698},
  archivePrefix = {arXiv},
  primaryClass = {cs},
  doi = {10.48550/arXiv.2605.01698},
  url = {https://arxiv.org/abs/2605.01698},
  note = {Preprint, submitted to Automation in Construction}
}

The v1 benchmark was introduced in:

@inproceedings{hellinNaturalLanguageInformation2025,
  title = {Natural {{Language Information Retrieval}} from {{BIM Models}}: {{An LLM-Based Agentic Workflow Approach}}},
  booktitle = {{{EC3}}},
  author = {Hellin, Sylvain and Nousias, Stavros and Borrmann, Andr{\'e}},
  year = 2025,
  doi = {http://www.doi.org/10.35490/EC3.2025.265},
  abstract = {While Building Information Models (BIM) effectively store building-related information, accessing it requires specialized software and expertise. Natural Language (NL) interfaces for BIM data retrieval can mitigate this challenge, but existing approaches are limited by rigid ontological frameworks or extensive pre-processing requirements. We present a Large Language Model-based agentic workflow that processes NL queries and automatically interacts with IFC-encoded BIM models without ontological or pre-processing constraints. In tests across architectural, structural, and MEP domains, our approach achieves 80\% overall accuracy. We provide open access to IFC-Bench-v1, our evaluation dataset containing various queries, answers, and reference BIM models.},
  copyright = {CC0 1.0 Universal Public Domain Dedication},
  langid = {english}
}

Acknowledgments

• —buildingSMART International for providing sample files
• —RWTH Aachen University E3D Institute for the Digital Hub models
• —KIT IAI for IFC example models
• —Tamira Wrabel, Sebastian Esser and Zijian Wang for preparing and curating the BIM Fundamentals SS2025 student models
• —TUM BIM Fundamentals SS2025 students for fantasy building models
• —Selahattin Doelger for the Sixty5 models
• —The openBIM community for quality assurance
• —Wawan Solihin (Singapore) for donating the West Riverside Hospital multi-discipline IFC models
• —Professor Robert Amor (University of Auckland) for hosting the models on the OpenIFC Model Repository

Fixed evaluation split (Hellin et al. 2026)

The main dataset is split-agnostic: questions/ifc-bench-v2.csv carries a stable id column (1..1027) as identity only, and defines no train/test partition.

For like-for-like comparison with the results reported in Hellin et al. 2026, a fixed evaluation split is published separately at questions/eval-split-hellin2026.csv (columns id,split). It pins the exact 514 held-out test questions (by id) used in that paper, with the remaining 513 questions as the train set. The split is seed-free and reproducible: join it to the dataset on id.

• —Test: 514 questions. Per category (1/2/3/4): 72 / 289 / 50 / 103.
• —Train: 513 questions.

This artifact is provided purely for reproducibility; the dataset itself remains split-agnostic, so other users are free to define their own splits over the id column.

Deduplication note (v2 update)

v2 now removes a duplicated question. One question ("Which door types have fire rating information?", category 2, project 4351) appeared twice, once in the train side and once in the test side of the fixed split. To avoid a train/test leak, the train-side copy (former id 21) was removed and the test-side copy (id 22) was retained.

id values are stable identifiers, not row positions, so surviving rows keep their ids and id 21 is now an intentional gap. The frozen 514-question evaluation split (questions/eval-split-hellin2026.csv) is unchanged: the same 514 test ids (including id 22) are pinned, and only the leaked train row was dropped, leaving 512 train questions.

The exact pre-dedup snapshot is preserved at git tag v2.0-frozen-split.

Maintainer: Sylvain Hellin | Contact: sylvain.hellin@tum.de | Issues: GitHub Issues