act13/AVA-Bench

AVA-Bench

Training dataset for the paper AVA-Bench: Atomic Visual Ability Benchmark for Vision Foundation Models (arXiv:2506.09082) accepted in CVPR 2026.

AVA-Bench is a diagnostic benchmark for evaluating Vision Foundation Models (VFMs) through Atomic Visual Abilities (AVAs): fundamental perceptual skills such as localization, counting, OCR, spatial understanding, depth estimation, color recognition, texture recognition, and fine-grained recognition.

AVA-Bench disentangls visual perception into 14 atomic visual capabilities, each with distribution-matched training and evaluation splits. This allows researchers to measure where a VFM excels or fails and to construct capability-level “ability fingerprints” for model comparison and selection.

This Hub release contains the training split of AVA-Bench. The evaluation split is released separately; see the project page and paper for details.

Capabilities

AVA-Bench covers 14 atomic visual capabilities, each released as its own subset/config:

Dataset structure

Each subset has a single train split, stored as Parquet shards with image bytes embedded in the file.

Data fields

Every example contains:

• —image (datasets.Image) — the input image, decoded as a PIL image on access.
• —id (string) — unique example identifier.
• —conversations (list of {from, value}) — instruction-tuning style turns. The human turn includes the question, usually with an <image> placeholder, and the gpt turn includes the ground-truth answer.

Some capabilities may additionally include fields such as:

• —height
• —width
• —category
• —area
• —bounding-box or region metadata, depending on the capability Per-subset row counts are visible in the dataset viewer's config dropdown.

Usage

Please go to github to use the dataset to evaluate Vision Foundation Models. If you want to check the dataset:

from datasets import load_dataset

# Load one capability
ds = load_dataset("act13/AVA-Bench", name="Counting", split="train")

print(ds[0])
# {
#   'id': '...',
#   'image': <PIL.Image.Image image mode=RGB ...>,
#   'conversations': [
#       {'from': 'human', 'value': '<image>\n...'},
#       {'from': 'gpt', 'value': '...'}
#   ],
#   ...
# }

To stream without downloading the full subset:

from datasets import load_dataset

ds = load_dataset(
    "act13/AVA-Bench",
    name="Counting",
    split="train",
    streaming=True,
)

for ex in ds.take(5):
    print(ex["conversations"][0]["value"])
    print(ex["conversations"][1]["value"])

Intended uses

AVA-Bench is intended for research on vision foundation models and vision-language systems. Suitable uses include:

• —Training or instruction-tuning vision-language models on atomic visual abilities.
• —Diagnosing which visual capabilities a VFM lacks.
• —Comparing VFMs through capability-level performance rather than only aggregate VQA accuracy.
• —Constructing balanced training mixtures across visual abilities.
• —Studying how different VFM pretraining objectives affect downstream perceptual capabilities.

Source datasets

AVA-Bench is curated from multiple existing datasets, depending on the atomic visual ability. Source datasets include, but are not necessarily limited to:

• —Objects365
• —LVIS
• —iNaturalist-2021
• —DIOR
• —NYU-Depth V2
• —KITTI
• —COCO-Text
• —IIIT5K
• —TextVQA
• —EgoOrientBench
• —CURE-OR
• —Places434
• —AID
• —CUB-200-2011
• —FGVC-Aircraft
• —MiT
• —DTD
• —Kylberg
• —KTH-TIPS
• —KTH-TIPS2

Please see the paper for the full per-capability dataset construction details and source-license breakdown.

License

This dataset card and the AVA-Bench organization/annotations are released under CC-BY-4.0.

Underlying images retain the licenses of their original source datasets. Users are responsible for respecting the license terms and usage restrictions of each source dataset.

Citation

If you use AVA-Bench, please cite:

@article{mai2025ava,
  title={Ava-bench: Atomic visual ability benchmark for vision foundation models},
  author={Mai, Zheda and Chowdhury, Arpita and Wang, Zihe and Jeon, Sooyoung and Wang, Lemeng and Hou, Jiacheng and Chao, Wei-Lun},
  journal={arXiv preprint arXiv:2506.09082},
  year={2025}
}

Contact

Open a discussion on this dataset's Community tab, or reach the authors via the contact information provided in the paper.