array/cola

COLA: Compose Objects Localized with Attributes

Self-contained Hugging Face port of the COLA benchmark from the paper "How to adapt vision-language models to Compose Objects Localized with Attributes?".

• —📄 Paper: https://arxiv.org/abs/2305.03689
• —🌐 Project page: https://cs-people.bu.edu/array/research/cola/
• —💻 Original code & data: https://github.com/ArijitRay1993/COLA

This repository bundles the benchmark annotations as Parquet files and the referenced images as regular files under images/, so the dataset is fully self-contained.

Dataset Structure

.
├── data/
│   ├── multiobjects.parquet
│   ├── singleobjects_gqa.parquet
│   ├── singleobjects_clevr.parquet
│   ├── singleobjects_paco.parquet
│   ├── singleobjects_gqa_labels.json
│   ├── singleobjects_clevr_labels.json
│   └── singleobjects_paco_labels.json
└── images/
    ├── vg/<vg_id>.jpg              # Visual Genome images (multiobjects + GQA)
    ├── clevr/valA/*.png            # CLEVR-CoGenT valA
    ├── clevr/valB/*.png            # CLEVR-CoGenT valB
    ├── coco/val2017/*.jpg          # COCO val2017 (PACO)
    └── coco/train2017/*.jpg        # COCO train2017 (PACO)

Image paths stored in parquet are relative to the repository root, e.g. images/vg/2390970.jpg. Load them by joining with the local clone / snapshot path.

Configs / Splits

multiobjects (210 pairs)

A hard image–caption matching task. Each row contains two images and two captions whose objects/attributes are swapped: caption 1 applies to image 1 (not image 2) and vice versa.

singleobjects_gqa (2,589 rows), singleobjects_clevr (30,000 rows), singleobjects_paco (7,921 rows)

Multi-label classification across fixed vocabularies of multi-attribute object classes (320 for GQA, 96 for CLEVR, 400 for PACO). The label lists live at data/singleobjects_<subset>_labels.json.

For a given class, the paper's MAP metric is computed on images where hard_list == 1 for that class. See scripts/eval.py in the original repo for the exact metric.

Loading

from datasets import load_dataset

mo   = load_dataset("array/cola", "multiobjects", split="val")
gqa  = load_dataset("array/cola", "singleobjects_gqa", split="val")
clv  = load_dataset("array/cola", "singleobjects_clevr", split="val")
paco = load_dataset("array/cola", "singleobjects_paco", split="val")

To open an image, resolve it against the local snapshot root:

from huggingface_hub import snapshot_download
from PIL import Image
import os

root = snapshot_download("array/cola", repo_type="dataset")
ex = mo[0]
img1 = Image.open(os.path.join(root, ex["image1"]))
img2 = Image.open(os.path.join(root, ex["image2"]))

Or, if you've cloned the repo with git lfs, just open paths directly:

Image.open(f"{REPO_DIR}/{ex['image1']}")

Licensing / Source notes

• —Visual Genome, CLEVR-CoGenT, and COCO images are redistributed here under their respective original licenses. Please refer to the upstream datasets:
• —Visual Genome (CC BY 4.0)
• —CLEVR-CoGenT (CC BY 4.0)
• —COCO 2017 (CC BY 4.0 for annotations; Flickr terms for images)
• —The COLA annotations (parquet files and label lists) are released under the MIT license, matching the original COLA repo.

Citation

@article{ray2023cola,
  title   = {COLA: How to adapt vision-language models to Compose Objects Localized with Attributes?},
  author  = {Ray, Arijit and Radenovic, Filip and Dubey, Abhimanyu and Plummer, Bryan A. and Krishna, Ranjay and Saenko, Kate},
  journal = {arXiv preprint arXiv:2305.03689},
  year    = {2023}
}