neurips2026-tdbench/tdbench-review

TDBench: Benchmarking Vision-Language Models on Top-Down Images

Note (Anonymous Review Version). This dataset card accompanies a NeurIPS 2026 Evaluations & Datasets double-blind submission. Author identifiers, institutional affiliations, project pages, and external repository links have been removed for the review period. The full set of public artifacts and the final citation will be restored upon decision.

Overview

TDBench is a benchmark for evaluating Vision-Language Models (VLMs) on top-down (also referred to as bird's-eye-view) imagery, with a focus on near-surface aerial and drone-style scenes. Top-down imagery exposes capabilities that are rarely stressed by front-view benchmarks, including reasoning about small objects, unusual perspectives, scale variation, and missing depth cues. TDBench combines a multiple-choice evaluation suite spanning multiple visual reasoning dimensions with a visual grounding component and a rotation-consistency protocol that exploits a key property of top-down scenes: physical meaning is largely preserved under in-plane rotation, even though VLM behavior is often not. The main benchmark contains 2,000 questions per rotation (1,800 multiple-choice + 200 visual grounding), evaluated at four rotation angles (0°, 90°, 180°, 270°).

Dataset Structure

The dataset is organized into three groups of resources:

• —Main multiple-choice splits (1,800 examples per split, available as Hugging Face splits):
• —main_0deg
• —main_90deg
• —main_180deg
• —main_270deg

• —Visual grounding files (200 examples per rotation, shipped as TSVs alongside the dataset):
• —tdbench_grounding_rot0.tsv
• —tdbench_grounding_rot90.tsv
• —tdbench_grounding_rot180.tsv
• —tdbench_grounding_rot270.tsv

Together with the multiple-choice splits, this yields 2,000 questions per rotation for the main benchmark (1,800 MCQ + 200 grounding).

• —Case-study splits (Hugging Face splits):
• —case_study_zoom_in
• —case_study_integrity
• —case_study_height
• —case_study_depth

Each multiple-choice example (main and case-study splits) contains the following fields:

Each visual grounding example (tdbench_grounding_rot*.tsv) contains the following fields:

The four main rotation splits and the four grounding TSVs each share the same underlying scenes per group; every scene appears once per rotation, with the image rotated by the indicated angle and answer choices or grounding targets transformed accordingly when the category requires it (e.g., directional questions, coordinate-bound answers).

Task Categories / Evaluation Dimensions

TDBench probes a range of capabilities relevant to top-down perception, including but not limited to:

• —Object recognition and presence detection
• —Object counting
• —Spatial reasoning (relative position, orientation, layout)
• —Scene and context understanding
• —Hallucination sensitivity
• —Rotation robustness (via RotationalEval; see below)
• —Visual grounding (200 grounding examples per rotation; evaluated through the downstream evaluation toolkit)

Each example carries a category label indicating which dimension it primarily targets.

RotationalEval

Standard evaluation reports per-example accuracy on a single rotation split (multiple-choice or grounding). RotationalEval is a complementary protocol that links the four rotations: a scene is counted as correct only if the model answers correctly on all four rotated versions of that scene, with answer choices or grounding targets appropriately transformed. RotationalEval applies independently to the multiple-choice splits (main_{0,90,180,270}deg) and to the grounding TSVs (tdbench_grounding_rot{0,90,180,270}).

Case Studies

The case-study splits target focused phenomena that frequently arise in top-down imagery:

1. 1.Zoom-in / digital magnification (`case_study_zoom_in`). Examines how magnifying or cropping affects recognition of small objects.
2. 2.Object integrity / partial occlusion (`case_study_integrity`). Examines behavior when objects are partially hidden, clipped, or occluded.
3. 3.Altitude / scale (`case_study_height`). Examines how varying capture altitude (and therefore object scale) affects recognition.
4. 4.Depth / Z-axis reasoning (`case_study_depth`). Examines depth-related reasoning from top-down views, in which depth cues are limited.

Usage with VLMEvalKit

TDBench is designed to be used with VLMEvalKit, a public VLM evaluation toolkit. After installing VLMEvalKit and configuring the desired model, the splits can be invoked through its standard run.py interface. The exact upstream installation instructions and integration links are intentionally omitted during the anonymous review period and will be restored after review.

Single-rotation evaluation:

python run.py --data tdbench_rot0 --model <model_name> --verbose --work-dir <results_directory>

RotationalEval over multiple-choice (run all four rotations together; the toolkit aggregates the rotation-consistent score automatically):

python run.py --data tdbench_rot0 tdbench_rot90 tdbench_rot180 tdbench_rot270 --model <model_name> --verbose --work-dir <results_directory>

Visual grounding (single rotation, with centroid-based judge):

python run.py --data tdbench_grounding_rot0 --model <model_name> --verbose --judge centroid --work-dir <results_directory>

RotationalEval over grounding (all four rotations):

python run.py --data tdbench_grounding_rot0 tdbench_grounding_rot90 tdbench_grounding_rot180 tdbench_grounding_rot270 --model <model_name> --verbose --judge centroid --work-dir <results_directory>

Case studies:

python run.py --data tdbench_cs_zoom tdbench_cs_height tdbench_cs_integrity tdbench_cs_depth --model <model_name> --verbose --work-dir <results_directory>

The identifiers tdbench_rot{0,90,180,270}, tdbench_grounding_rot{0,90,180,270}, and tdbench_cs_{zoom,height,integrity,depth} correspond to the resources described in this dataset card.

Expected Outputs

VLMEvalKit prints and saves each dataset's output under <results_directory>/<model_name>/. For each evaluated split, the toolkit produces:

• —*_acc.csv — per-category and overall accuracy.
• —*_result.xlsx — detailed per-example model outputs and judging results.

When all four main rotation splits are evaluated for the same model, RotationalEval is triggered automatically and its rotation-consistent results are written to *_REresult.csv.

Intended Use

TDBench is intended for research-oriented evaluation of vision-language models on near-surface top-down or bird's-eye-view imagery. Suitable use cases include:

• —Object recognition and counting in top-down scenes.
• —Spatial reasoning and scene understanding.
• —Hallucination analysis and reliability studies.
• —Visual grounding on top-down imagery (200 grounding examples per rotation, via the downstream evaluation toolkit).
• —Rotation-robustness evaluation via RotationalEval (applicable to both multiple-choice and grounding).

Out-of-Scope Use

TDBench should not be used for, or to support claims about:

• —Surveillance or person tracking.
• —Identification of individuals, including face recognition.
• —License-plate recognition.
• —Fine-grained geolocation of scenes or subjects.
• —Inference of demographic, biometric, or other personal attributes.
• —Safety-critical operational decisions (e.g., live drone navigation, autonomous flight control, emergency response automation).
• —Broad claims about general VLM competence beyond the specific capabilities and scenes represented here.

Limitations

TDBench is a curated, benchmark-style evaluation set and does not exhaustively cover all real-world top-down imagery — for example, all possible geographic regions, capture altitudes, sensor modalities, or application domains. To broaden coverage of weather and lighting conditions in particular, a subset of the questions use imagery rendered from simulation environments to mimic conditions that are difficult to source at scale from real captures.

Biases

The benchmark may reflect selection and distributional biases inherited from its source imagery and annotation process, including:

• —Scene-type bias: urban, traffic, sports, infrastructure, or open-area scenes may be overrepresented relative to other aerial applications.
• —Object-frequency bias: some object types appear more often than others.
• —Geographic and environmental bias: source imagery may not evenly cover regions, seasons, weather, lighting, or built environments.
• —Capture-condition bias: altitude, camera angle, resolution, and sensor characteristics may vary across sources.
• —Annotation-design bias: question templates and answer choices may emphasize some reasoning skills more than others.
• —Source-dataset selection bias inherited from upstream imagery sources.

Personal or Sensitive Information

TDBench is not designed to identify individuals or to infer personal attributes. Annotations do not include names, personal identifiers, faces, license plates, or demographic labels. Because the underlying imagery is top-down and drawn from publicly available sources, some images may incidentally contain small visible people or vehicles; the benchmark does not target, label, or evaluate these elements as identifiers, and users should not repurpose the data for identification tasks.

Synthetic Data

TDBench does not contain images produced by generative models. A subset of questions uses simulation-rendered imagery to control weather, lighting, or viewpoint conditions that are difficult to source at scale from real captures. We therefore distinguish between (i) generative-model-produced synthetic images, which are not used, and (ii) simulation-rendered imagery, which is included for controlled evaluation. Rotated images and transformed labels are deterministic derived data.

Data Provenance

The full provenance of the source imagery and annotation pipeline is described explicitly in the accompanying paper. At a high level, imagery is drawn from publicly available top-down/aerial image sources (and, for a subset of questions, from simulation environments), with question–answer pairs and grounding targets manually curated by the dataset creators.

License

This dataset is released under CC BY-NC-SA 4.0, as indicated in the dataset card metadata. Users must comply with the terms of this license, as well as with the licenses and terms of use of the upstream source datasets from which imagery was derived.

Citation (Anonymized During Review)

@misc{anonymous2026tdbench,
  title={TDBench: Benchmarking Vision-Language Models on Top-Down Images},
  author={Anonymous Authors},
  year={2026},
  note={Submitted for double-blind review}
}