array/cola

1---2license: mit3task_categories:4- image-classification5- image-to-text6- zero-shot-image-classification7language:8- en9pretty_name: COLA10size_categories:11- 10K<n<100K12tags:13- compositionality14- vision-language15- visual-genome16- clevr17- paco18configs:19- config_name: multiobjects20  data_files:21  - split: val22    path: data/multiobjects.parquet23- config_name: singleobjects_gqa24  data_files:25  - split: val26    path: data/singleobjects_gqa.parquet27- config_name: singleobjects_clevr28  data_files:29  - split: val30    path: data/singleobjects_clevr.parquet31- config_name: singleobjects_paco32  data_files:33  - split: val34    path: data/singleobjects_paco.parquet35---36 37# COLA: Compose Objects Localized with Attributes38 39Self-contained Hugging Face port of the **COLA** benchmark from the paper40["How to adapt vision-language models to Compose Objects Localized with Attributes?"](https://arxiv.org/abs/2305.03689).41 42- 📄 Paper: https://arxiv.org/abs/2305.0368943- 🌐 Project page: https://cs-people.bu.edu/array/research/cola/44- 💻 Original code & data: https://github.com/ArijitRay1993/COLA45 46This repository bundles the benchmark annotations as Parquet files and the referenced47images as regular files under `images/`, so the dataset is fully self-contained.48 49## Dataset Structure50 51```52.53├── data/54│   ├── multiobjects.parquet55│   ├── singleobjects_gqa.parquet56│   ├── singleobjects_clevr.parquet57│   ├── singleobjects_paco.parquet58│   ├── singleobjects_gqa_labels.json59│   ├── singleobjects_clevr_labels.json60│   └── singleobjects_paco_labels.json61└── images/62    ├── vg/<vg_id>.jpg              # Visual Genome images (multiobjects + GQA)63    ├── clevr/valA/*.png            # CLEVR-CoGenT valA64    ├── clevr/valB/*.png            # CLEVR-CoGenT valB65    ├── coco/val2017/*.jpg          # COCO val2017 (PACO)66    └── coco/train2017/*.jpg        # COCO train2017 (PACO)67```68 69Image paths stored in parquet are **relative to the repository root**, e.g.70`images/vg/2390970.jpg`. Load them by joining with the local clone / snapshot path.71 72## Configs / Splits73 74### `multiobjects` (210 pairs)75 76A hard image–caption matching task. Each row contains two images and two captions77whose objects/attributes are swapped: caption 1 applies to image 1 (not image 2) and78vice versa.79 80| Field      | Type   | Description                       |81|------------|--------|-----------------------------------|82| `image1`   | string | Relative path to image 1          |83| `caption1` | string | Caption describing image 1        |84| `image2`   | string | Relative path to image 2          |85| `caption2` | string | Caption describing image 2        |86 87### `singleobjects_gqa` (2,589 rows), `singleobjects_clevr` (30,000 rows), `singleobjects_paco` (7,921 rows)88 89Multi-label classification across fixed vocabularies of multi-attribute object90classes (320 for GQA, 96 for CLEVR, 400 for PACO). The label lists live at91`data/singleobjects_<subset>_labels.json`.92 93| Field                | Type            | Description                                                   |94|----------------------|-----------------|---------------------------------------------------------------|95| `image`              | string          | Relative path to the image                                    |96| `objects_attributes` | string (JSON)   | Objects + attributes annotation (GQA and CLEVR only)          |97| `label`              | list\[int]      | Binary indicator per class (length matches labels vocabulary) |98| `hard_list`          | list\[int]      | Indicator of whether each class is "hard" for this image      |99 100For a given class, the paper's MAP metric is computed on images where `hard_list == 1`101for that class. See `scripts/eval.py` in the [original repo](https://github.com/ArijitRay1993/COLA)102for the exact metric.103 104## Loading105 106```python107from datasets import load_dataset108 109mo   = load_dataset("array/cola", "multiobjects", split="val")110gqa  = load_dataset("array/cola", "singleobjects_gqa", split="val")111clv  = load_dataset("array/cola", "singleobjects_clevr", split="val")112paco = load_dataset("array/cola", "singleobjects_paco", split="val")113```114 115To open an image, resolve it against the local snapshot root:116 117```python118from huggingface_hub import snapshot_download119from PIL import Image120import os121 122root = snapshot_download("array/cola", repo_type="dataset")123ex = mo[0]124img1 = Image.open(os.path.join(root, ex["image1"]))125img2 = Image.open(os.path.join(root, ex["image2"]))126```127 128Or, if you've cloned the repo with `git lfs`, just open paths directly:129 130```python131Image.open(f"{REPO_DIR}/{ex['image1']}")132```133 134## Licensing / Source notes135 136- Visual Genome, CLEVR-CoGenT, and COCO images are redistributed here under their137  respective original licenses. Please refer to the upstream datasets:138  - [Visual Genome](https://visualgenome.org/) (CC BY 4.0)139  - [CLEVR-CoGenT](https://cs.stanford.edu/people/jcjohns/clevr/) (CC BY 4.0)140  - [COCO 2017](https://cocodataset.org/) (CC BY 4.0 for annotations; Flickr terms for images)141- The COLA annotations (parquet files and label lists) are released under the MIT142  license, matching the [original COLA repo](https://github.com/ArijitRay1993/COLA).143 144## Citation145 146```bibtex147@article{ray2023cola,148  title   = {COLA: How to adapt vision-language models to Compose Objects Localized with Attributes?},149  author  = {Ray, Arijit and Radenovic, Filip and Dubey, Abhimanyu and Plummer, Bryan A. and Krishna, Ranjay and Saenko, Kate},150  journal = {arXiv preprint arXiv:2305.03689},151  year    = {2023}152}153```154