EmbodiedCity/BasicSpatialAbility
[ACL'25 Main] Defining and Evaluating Visual Language Models’ Basic Spatial Abilities: A Perspective from Psychometrics [!IMPORTANT] You can find the sample testing code on GitHub! This dataset is a benchmark designed for evaluating Multimodal Large Language Models' Basic Spatial Abilities based on authentic Psychometric theories. It is structured specifically to support both Zero-shot and Few-shot evaluation protocols. Split Name Role Description test Query Set… See the full description on the dataset page: https://huggingface.co/datasets/EmbodiedCity/BasicSpatialAbility.
0238
1---2license: mit3task_categories:4- visual-question-answering5language:6- en7tags:8- video9- text10- embodied11- spatial12- multimodal13size_categories:14- n<1K15---16 17# [ACL'25 Main] Defining and Evaluating Visual Language Models’ Basic Spatial Abilities: A Perspective from Psychometrics18[](https://aclanthology.org/2025.acl-long.567/)19[](https://arxiv.org/abs/2502.11859)20[](https://github.com/EmbodiedCity/BasicSpatialAbility.code)21[](https://huggingface.co/datasets/EmbodiedCity/BasicSpatialAbility)22 23> [!IMPORTANT]24> **You can find the sample testing code on GitHub!**25 26This dataset is a benchmark designed for evaluating Multimodal Large Language Models' Basic Spatial Abilities based on authentic Psychometric theories. It is structured specifically to support both **Zero-shot** and **Few-shot** evaluation protocols.27 28| Split Name | Role | Description |29| :--- | :--- | :--- |30| **`test`** | **Query Set** | Contains the actual benchmark questions (images & queries) to be evaluated. <br>⚠️ **Evaluation Only.** Do not use for training or as few-shot examples. |31| **`validation`** | **Support Set** | Contains high-quality examples intended to be used as **Few-shot Prompts (In-Context Learning)**. <br>These samples should be prepended to the test queries to demonstrate the task to the model. |32 33# ⚙️ Usage & Evaluation Protocol34You can load the dataset using the Hugging Face `datasets` library.35 36### 1. Zero-Shot Evaluation37**Logic:** Directly evaluate the model on the `test` split without any prior examples.38 39```python40from datasets import load_dataset41 42# Load the evaluation queries43test_dataset = load_dataset("EmbodiedCity/BasicSpatialAbility", split="test")44 45for sample in test_dataset:46 image = sample['image']47 question = sample['question']48 # Model inference...49```50 51### 2. Few-Shot Evaluation52**Logic:** Use examples from the validation split as the context (demonstrations), followed by the query from the test split.531. Load the validation split.542. Format them into the prompt history.553. Append the target question from the test split.56 57```python58from datasets import load_dataset59 60# 1. Load the support set (demonstrations)61support_set = load_dataset("EmbodiedCity/BasicSpatialAbility", split="validation")62 63# 2. Load the query set (evaluation)64test_set = load_dataset("EmbodiedCity/BasicSpatialAbility", split="test")65 66# Pseudo-code for prompt construction67prompt_context = []68for ex in support_set:69 prompt_context.append(f"User: {ex['question']}\nAssistant: {ex['answer']}")70 71# 3. Evaluate on Test Set72for sample in test_set:73 # Combine context + current test question74 final_prompt = prompt_context + [f"User: {sample['question']}"]75 76 # Model inference...77```78 79# 🔬 Underlying Theory80The Theory of Multiple Intelligences underscores the hierarchical nature of cognitive capabilities. To advance Spatial Artificial Intelligence, we pioneer a psychometric framework defining five Basic Spatial Abilities (BSAs) in Visual Language Models (VLMs): Spatial Perception, Spatial Relation, Spatial Orientation, Mental Rotation, and Spatial Visualization. Benchmarking 13 mainstream VLMs through nine validated psychometric experiments reveals significant gaps versus humans, with three key findings: 1) VLMs mirror human hierarchies (strongest in 2D orientation, weakest in 3D rotation) with independent BSAs; 2) Many smaller models surpass larger counterparts, with Qwen leading and InternVL2 lagging; 3) Interventions like CoT and few-shot training show limits from architectural constraints, while ToT demonstrates the most effective enhancement. Identified barriers include weak geometry encoding and missing dynamic simulation. By linking Psychometrics to VLMs, we provide a comprehensive BSA evaluation benchmark, a methodological perspective for embodied AI development, and a cognitive science-informed roadmap for achieving human-like spatial intelligence.81| Type | Definition | Tests |82|:----------------------:|:---------------------------------------------------------------------------------------------------------------------:|:-----------------------:|83| Spatial Perception | The ability to perceive horizontal and vertical orientations without interference from miscellaneous information. | SVT |84| Spatial Relation | The ability of recognizing relationships between parts of an entity. | NCIT DAT:SR R-Cube-SR |85| Spatial Orientation | The ability to navigate or enter a given spatial state. | MRMT |86| Mental Rotation | The ability to mentally rotate 3D objects. | MRT PSVT:R |87| Spatial Visualization | The ability to mentally manipulate and transform 2D and 3D objects. | SBST R-Cube-Vis |****88 89<p align="center">90 <img width="600" src="https://github.com/EmbodiedCity/BasicSpatialAbility.code/raw/main/framework.jpg">91</p>92 93<p align="center">94 The Framework of Basic Spatial Abilities (Image sources are cited in the paper)95</p>96 97# Citation98If you use this project in your research, please cite the following paper:99```bibtex100@inproceedings{xu-etal-2025-defining,101 title = "Defining and Evaluating Visual Language Models' Basic Spatial Abilities: A Perspective from Psychometrics",102 author = "Xu, Wenrui and103 Lyu, Dalin and104 Wang, Weihang and105 Feng, Jie and106 Gao, Chen and107 Li, Yong",108 booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",109 month = jul,110 year = "2025",111 address = "Vienna, Austria",112 publisher = "Association for Computational Linguistics",113 url = "https://aclanthology.org/2025.acl-long.567/",114 doi = "10.18653/v1/2025.acl-long.567",115 pages = "11571--11590",116 ISBN = "979-8-89176-251-0"117}