ylsung/VL-Adapter-datasets

VL-Adapter Datasets

Processed CLIP-ResNet101 grid features and annotations for **VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks** (Sung, Cho, Bansal — CVPR 2022), code at ylsung/VL_adapter.

This repository replaces the original Google Drive download, which is no longer available. It holds the same data in a Hub-native layout, plus a script that rebuilds the exact datasets/ directory tree the training code expects.

Quick start — rebuild the original datasets/ tree

pip install huggingface_hub pyarrow h5py numpy
wget https://huggingface.co/datasets/ylsung/VL-Adapter-datasets/raw/main/restore_datasets.py
python restore_datasets.py --out ./datasets

This downloads one shard at a time and deletes it after expanding, so it needs less than ~1 GB of scratch space beyond the final tree. It is resumable — rerun it and it skips what is already on disk. Expect roughly 145 GB for the restored tree (the .h5 files are uncompressed) and ~49 GB of download.

To restore a subset:

python restore_datasets.py --out ./datasets --datasets GQA --datasets nlvr
python restore_datasets.py --out ./datasets --only annotations

Using the features directly

If you just want the features and not the VL-Adapter directory layout:

from datasets import load_dataset

ds = load_dataset("ylsung/VL-Adapter-datasets", "GQA", split="train", streaming=True)
row = next(iter(ds))
row["id"]        # '2336439'  -> the image id
row["features"]  # (49, 2048) float16 CLIP-RN101 attention-pool grid features

Contents

Feature schema

Each row is one image:

In the original release each of these rows was a standalone .h5 file holding a single group named id with one (49, 2048) float16 dataset called features. 621,783 such files exceed the Hub's recommended limits (<100k files per repo, <10k per folder), so they are packed into ~150 zstd-compressed parquet shards of 4,096 rows each. restore_datasets.py unpacks them back to identical .h5 files — the arrays it writes are bit-exact and the files are the same 203,136 bytes; only an internal HDF5 heap offset differs.

Annotations

annotations/ mirrors the original folder layout byte-for-byte (annotations/vqa/karpathy_train.json → datasets/vqa/karpathy_train.json). They are left as raw JSON rather than converted to parquet because their schemas vary widely and the training code parses them directly.

annotations/GQA/paragraph_testdev.json is 0 bytes, and COCO/clip_features/data_clip_RN101_fc/ and paragraphs/ are empty, in the original release too.

NLVR2 images are not included

The original archive also contained 119,354 NLVR2 photographs. The NLVR2 authors do not own their copyright and ask that they not be shared publicly, so only the CLIP features derived from them are published here.

Every script under VL-T5/scripts/image/ reads clip_features and runs without the images. They are needed only for the end-to-end pixel training paths (VL-T5/src/*_raw_data.py). Request them from the NLVR2 repository and place them at datasets/nlvr/images/.

Raw COCO, GQA and Visual Genome images are likewise not included; get them from COCO, GQA and Visual Genome.

Extracting your own features

See `feature_extraction/` in the code repository to build features for other CLIP backbones.

Licensing

The packaging code and the VL-Adapter project are MIT licensed. The underlying annotations and images keep the licenses of their original sources — MS COCO, Visual Genome, GQA, VQA v2 and NLVR2 — and the derived features are released for research use on the same terms.

Citation

@inproceedings{sung2022vladapter,
  title     = {VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks},
  author    = {Yi-Lin Sung, Jaemin Cho, Mohit Bansal},
  booktitle = {CVPR},
  year      = {2022}
}