mapspatial/map-spatial-benchmark

Map-based Spatial Reasoning Benchmark

A multi-view map-based spatial reasoning benchmark. Each row is one multiple-choice question instance over a registered map image; models must answer with a single option letter. Four tasks (T1–T4), four base-map views, and controlled evidence conditions (direct / query / oracle) and world perturbations (transform / world layers) allow fine-grained analysis of spatial reasoning robustness.

Task overview

Supervision conditions

Each task is released under one or more supervision conditions, which differ only in the visual evidence overlaid on the registered map images — the questions, options and answers are identical across conditions of the same task.

Note: for T4 detour_waypoint_traversability, the query condition uses the fully annotated road-status / waypoint-map pair, since the weak scaffold does not cover this question type.

Sample counts (per condition)

T1/T2 provide 2 conditions each; T3/T4 provide 3. All conditions of a task share the same instance set. Total: 77,800 rows (40,000 for T1/T2 + 13,800 for T3 + 24,000 for T4).

T1 — 10,000 per condition

Base question set = 400 instances per view. 7 geometric transform variants (rot90, rot180, rot270, mirrorh, mirrorh_rot90/180/270) replicate the base set.

T2 — 10,000 per condition

Same structure as T1 (400 base instances, same view/transform coverage): 3,200 x 3 + 400 = 10,000.

T3 — 4,600 per condition

Base question set = 200 instances per view; world layer has 2 variants (intervention_001, sham_001) with 200 each. The blank view is not used for T3: the task is building-footprint recognition, which is only meaningful on imagery that actually shows buildings (satellite / map tiles).

T4 — 8,000 per condition

Base question set = 400 instances; world layer = 2 variants x 200. The blank view is not used for T4: the task is road/waypoint recognition on the route network, which requires imagery that actually depicts roads (satellite / map tiles).

Dimensions

Row schema

{
  "id": "c0019932db15_blank_query",
  "task_id": "T1",
  "layer": "base",
  "variant": null,
  "view": "blank",
  "condition": "query",
  "question_type": "angular_order",
  "question": "From the green point, sweeping clockwise from north, ...",
  "options": ["A. yellow, red", "B. red, yellow"],
  "answer": "B",
  "answer_letter": "B",
  "system_prompt": "You are evaluating spatial reasoning ...",
  "images": ["images/t1/c_06d46000d7/s_429d01__q_e5ebc72207__base__query__blank_query_q4.png"],
  "meta": {
    "case_id": "c_06d46000d7",
    "scheme": "s_429d01",
    "instance_id": "q_e5ebc72207",
    "tile_type": "blank",
    "subtask": null
  }
}

• —options are pre-shuffled; the correct letter is answer_letter.
• —Evaluation is exact match on the option letter for all rows.
• —Multiple-choice prompt format: append Options:\n<options> and instruct the model to output only the letter (see system_prompt).

Image naming

Images live under images/{task}/{case_code}/. Directory segments case_id, scheme and question_dir are anonymized with deterministic short codes (c_*, s_*, q_*); the remaining segments (layer, condition, original file name) are kept verbatim, e.g.

images/t1/c_06d46000d7/s_429d01__q_e5ebc72207__base__query__blank_query_q4.png

The same source image may be referenced by several rows (e.g., shared across transform variants or views); 77,800 rows reference 27,394 unique image files.

Loading

from datasets import load_dataset
from huggingface_hub import snapshot_download
from PIL import Image

# Download the image files once (the jsonl only stores relative image paths)
root = snapshot_download("mapspatial/map-spatial-benchmark", repo_type="dataset")

ds = load_dataset("mapspatial/map-spatial-benchmark", "t1_query_supervision", split="test")
row = ds[0]
img = Image.open(f"{root}/{row['images'][0]}")

Anonymization note

Identifiers of the source map data (case / scheme / question directory) are replaced by deterministic hash codes and are not reversible from this repository.