turhancan97/SpaRRTa

<h1 style="display:flex; align-items:center; gap:10px;"> <img src="assets/logo.png" alt="Logo" width="55"> <span style="color:#FF7096;">SpaRRTa</span>: A Synthetic Benchmark for Evaluating Spatial Intelligence in Visual Foundation Models </h1>

SpaRRTa is a synthetic benchmark that probes whether Visual Foundation Models (VFMs) — such as DINO, DINOv2/v3, MAE, CroCo, VGGT, SPA and CLIP — encode the spatial relations between objects in a scene, rather than only their semantic identity.

• —📄 Paper: arXiv:2601.11729
• —💻 Code: github.com/gmum/SpaRRTa
• —🧱 Real-world (lego) split: turhancan97/SpaRRTa-Lego
• —🔬 Attention-analysis split (images + masks): turhancan97/SpaRRTa-Attention

This repository hosts the synthetic (Unreal Engine 5) portion of the benchmark. It is complemented by two companion splits: the real-world set photographed with toy minifigures for sim-to-real evaluation (`turhancan97/SpaRRTa-Lego`), and an attention-analysis set with per-object segmentation masks (`turhancan97/SpaRRTa-Attention`).

<p align="center"> <img src="assets/teaser.png" alt="SpaRRTa teaser" width="100%"> </p>

The task

SpaRRTa is a 4-way classification problem — Front / Back / Left / Right — asking where a target object lies relative to a reference object, from a given viewpoint. It has two variants:

• —SpaRRTa-ego (egocentric): directions are defined from the camera's viewpoint.
• —SpaRRTa-allo (allocentric): directions are defined from a human figure's viewpoint in the scene, which requires implicit perspective-taking.

Both variants use the same images: each sample's params_*.json stores the 3D positions of the camera, the human, and the scene objects, so the egocentric/allocentric label is computed at load time by choosing the observer (camera vs. human). Direction labels are therefore derived from geometry, not stored as a column — the dataset provides the raw positions and the code turns them into Front/Back/Left/Right labels (excluding configurations within ±15° of a diagonal boundary as ambiguous).

Note: the actor_labels column lists object identities (e.g. Human, Tree, Truck), not the Front/Back/Left/Right classes. Which object is the reference and which is the target is an experiment setting defined per environment in the code.

Environments

Rendered in Unreal Engine 5 (≈2048×2048 px) across five environments, each with three variants: bridge, city, desert, forest, winter_town.

The overall data-generation and probing pipeline is summarized below:

<p align="center"> <img src="assets/pipeline.png" alt="SpaRRTa data-generation and probing pipeline" width="100%"> </p>

Dataset statistics

• —Total samples: 149,145 (single train split)
• —Broken / missing pairs: 0
• —Splits: shipped as train only; train/validation/test partitions are produced deterministically in the training code from a fixed seed (so results are reproducible from this single split).

Scene coverage

Columns

• —sample_id (string): stable unique id (scene_variant:frame_id)
• —scene (string): base scene name (e.g. bridge)
• —variant (int): numeric variant from folder suffix (bridge_3 → 3, base folder → 1)
• —scene_variant (string): source folder name (this is the value used as environment= in the code)
• —frame_id (int): numeric frame id from filename
• —image (image): rendered RGB frame (embedded bytes + relative path)
• —image_relpath (string): relative source image path
• —params_relpath (string): relative source JSON path
• —raw_params_json (string): full original JSON text (camera, actors, source)
• —camera_json (string): camera section (location, rotation, intrinsics)
• —actors_json (string): actors section (per-object label + 3D location)
• —source_json (string): source section
• —actor_labels (list[string]): unique object identities found in actors
• —has_label_mapping (bool): whether source.label_mapping exists
• —label_mapping_json (string): full mapping JSON
• —label_mapping_keys (list[string]): mapping keys
• —label_mapping_values (list[string]): mapping values
• —original_params_name (string): source.original_params when present
• —upload_batch_utc (string): UTC timestamp of upload run

Loading

from datasets import load_dataset

ds = load_dataset("turhancan97/SpaRRTa", split="train")
print(ds[0]["scene_variant"], ds[0]["actor_labels"])
ds[0]["image"]  # PIL.Image (decoded automatically)

Download to a local machine

huggingface-cli download turhancan97/SpaRRTa --repo-type dataset --local-dir ./hf_SpaRRTa

Use with the SpaRRTa code

The training code reads images and annotations from disk under $SPARRTA_DATA_ROOT/<environment>/mid-objects/. The snippet below reconstructs exactly that layout from the parquet shards:

from pathlib import Path
from datasets import load_dataset, Image

repo_id = "turhancan97/SpaRRTa"
output_root = Path("position_between_objects")
output_root.mkdir(parents=True, exist_ok=True)

ds = load_dataset(repo_id, split="train")
ds = ds.cast_column("image", Image(decode=False))  # keep raw bytes

for row in ds:
    # Reconstruct <output>/<scene_variant>/mid-objects/img_XXXX.jpg + params_XXXX.json
    mid = output_root / row["scene_variant"] / "mid-objects"
    mid.mkdir(parents=True, exist_ok=True)

    image_name = Path(row["image_relpath"]).name
    params_name = Path(row["params_relpath"]).name

    image_bytes = row["image"]["bytes"]
    if image_bytes is None:
        raise RuntimeError(f"Missing embedded image bytes for {row['sample_id']}")

    (mid / image_name).write_bytes(image_bytes)
    (mid / params_name).write_text(row["raw_params_json"], encoding="utf-8")

Then point the code at the reconstructed folder and train a probe (e.g. egocentric forest with DINO + EfficientProbing):

export SPARRTA_DATA_ROOT=$(pwd)/position_between_objects
python train.py \
  backbone=dino_b16 \
  dataset=unreal_position \
  probe=classifier probe._target_=sparrta.models.probes.EfficientProbing \
  dataset.perspective=camera \
  environment=forest

Use dataset.perspective=human for the allocentric task, and any scene_variant above as the environment= value. See the code repository for full instructions, backbones, and probing heads.

License

Released under the MIT License.

Citation

If you find this dataset useful, please consider citing:

@misc{kargin2026sparrta,
  title={SpaRRTa: A Synthetic Benchmark for Evaluating Spatial Intelligence in Visual Foundation Models},
  author={Turhan Can Kargin and Wojciech Jasiński and Adam Pardyl and Bartosz Zieliński and Marcin Przewięźlikowski},
  year={2026},
  eprint={2601.11729},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2601.11729}
}