Datasetpublic

earth-insights/OVEarth-Bench

OVEarth-Bench OVEarth-Bench is an evaluation benchmark for open-vocabulary Earth-observation image understanding. It evaluates whether a model can recognize, segment, and localize remote-sensing targets from category names, referring expressions, and reasoning-oriented queries. The benchmark additionally includes negative open-vocabulary queries to measure hallucination suppression. The associated evaluation toolkit is available at earth-insights/OVEarth-bench.… See the full description on the dataset page: https://huggingface.co/datasets/earth-insights/OVEarth-Bench.

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

3likes106downloads

Dataset Card

OVEarth-Bench

OVEarth-Bench is an evaluation benchmark for open-vocabulary Earth-observation image understanding. It evaluates whether a model can recognize, segment, and localize remote-sensing targets from category names, referring expressions, and reasoning-oriented queries. The benchmark additionally includes negative open-vocabulary queries to measure hallucination suppression.

The associated evaluation toolkit is available at earth-insights/OVEarth-bench.

Dataset Summary

Item	Value
Images	520 newly collected EO GeoTIFF images (`.tif`)
Masks	590 human-verified masks
Horizontal boxes	4,810 axis-aligned boxes (`bbox`)
Oriented boxes	4,810 oriented boxes (`oriented_bbox`)
Categories	172 categories across seven EO domains
Vocabulary	1,346 unique vocabulary strings
Queries	5,023 English text queries
Annotation modalities	COCO RLE masks, axis-aligned boxes, oriented boxes
Image dimensions	Widths range from 692 to 4,713 pixels; median GSD is 0.30 m/pixel
License	Apache-2.0

Tasks

Task key	Task	Queries	Notes
`open_vocabulary`	Open-vocabulary segmentation and detection	3,235	1,067 positive and 2,168 negative category phrases. Negative queries evaluate false-positive suppression.
`referring`	Referring-expression segmentation and grounding	732	Spatial or contextual descriptions of a target.
`reasoning`	Reasoning segmentation and grounding	1,056	Functional or attribute-based descriptions requiring target inference.

The benchmark provides ground-truth segmentation masks, axis-aligned boxes, and oriented boxes for all three tasks where the corresponding annotation is available.

Files

File	Description
`images.zip`	Archive containing the GeoTIFF images referenced by `image_path` in the annotations.
`ground_truth.json`	Ground truth instances and task queries.
`README.md`	This dataset card.

After extracting images.zip, preserve the images/ directory structure so that paths in ground_truth.json remain valid.

Data Collection and Quality Control

All images were newly collected rather than copied from existing remote-sensing benchmarks. POI-guided retrieval samples six continents under varied scales and seasonal conditions. The taxonomy combines national standards, more than 200 datasets, and OpenStreetMap tags.

Annotators draw or correct instance polygons, and managers review boundaries and omissions. Horizontal and minimum-area oriented boxes are derived from reviewed instance polygons. Language queries are generated with LLM assistance, then subjected to automated checks and expert review.

Annotation Format

ground_truth.json has two top-level keys: annotations and tasks. A task query links to an annotation through ann_id.

`annotations`

Field	Type	Description
`id`	string	Unique annotation identifier.
`image_path`	string	Relative path to the GeoTIFF image, for example `images/1.tif`.
`width`, `height`	integer	Image width and height in pixels.
`category_name`	string	Canonical English category name.
`segmentation`	object	COCO RLE mask with `size: [height, width]` and `counts`.
`bbox`	list or `null`	Axis-aligned ground-truth boxes in COCO `[x, y, width, height]` (`xywh`) format.
`oriented_bbox`	list or `null`	Oriented boxes represented by four image-coordinate vertices.

segmentation uses COCO RLE. size is [height, width], and the mask is encoded in column-major (Fortran) order. The current release stores compressed RLE strings in counts.

`tasks`

Field	Used by	Description
`query_id`	all tasks	Unique query identifier.
`ann_id`	all tasks	ID of the linked entry in `annotations`.
`type`	`open_vocabulary`	`positive` when the query target is present; `negative` when it is absent.
`phrase`	`open_vocabulary`	Category phrase supplied to the model.
`query`	`referring`, `reasoning`	Referring or reasoning-oriented natural-language query.

For a negative open-vocabulary query, the linked annotation may describe another object in the image and must not be used as geometry for the queried target.

Download

Install the ModelScope SDK and download the dataset repository:

bash

pip install modelscope
modelscope download earth-insights/OVEarth-bench --repo-type dataset

Alternatively, download individual files from the Files and versions tab. Extract images.zip before running an evaluation.

Evaluation

The evaluation toolkit and usage instructions are available at earth-insights/OVEarth-bench.

Citation

If you use this dataset, please cite:

bibtex

@article{li2026ovearth,
  title   = {OVEarth-Bench: Evaluating Category Breadth and
             Query Diversity for Open-Vocabulary Earth Observation},
  author  = {Li, Kaiyu and Xin, Zepeng and Jiang, Zixuan
             and Fu, Jing and Xue, Lanxuan and Zhang, Lingyu
             and Cao, Xiangyong},
  journal = {arXiv preprint arXiv:2607.27278},
  year    = {2026}
}