Datasetpublic

freek23/cc3m-grounded-annotations

CC3M grounded annotations Region-level grounding for Conceptual Captions 3M: bounding boxes, the noun phrase each box grounds, and the span of the caption that phrase came from, for 3,016,640 of CC3M's 3,318,333 rows. No images here. This is metadata only, joinable onto a CC3M copy you already have. That is the point of it: the grounding is 354 MB, the pixels are 125 GB. Files file rows size annotations-0000..0482.parquet 3,016,640 199 MB… See the full description on the dataset page: https://huggingface.co/datasets/freek23/cc3m-grounded-annotations.

sourceHugging Faceotherupdated 2mo agoView on Hugging Face

0likes48downloads

Dataset Card

CC3M grounded annotations

Region-level grounding for Conceptual Captions 3M: bounding boxes, the noun phrase each box grounds, and the span of the caption that phrase came from, for 3,016,640 of CC3M's 3,318,333 rows.

No images here. This is metadata only, joinable onto a CC3M copy you already have. That is the point of it: the grounding is 354 MB, the pixels are 125 GB.

Files

file	rows	size
`annotations-0000..0482.parquet`	3,016,640	199 MB
`cc3m_row_map.parquet`	3,016,640	155 MB

`annotations-*.parquet`

column	type	meaning
`data_id`	int64	GLIGEN's sample id, the primary key
`caption`	string	the CC3M caption
`width`, `height`	int32	dimensions the boxes are expressed in
`boxes_xywh`	list of 4 int32	boxes as x, y, w, h in pixels of `width` x `height`
`phrases`	list of string	the phrase each box grounds, aligned with `boxes_xywh`
`tokens_positive`	string	JSON, character spans of each phrase within `caption`

`cc3m_row_map.parquet`

column	type	meaning
`data_id`	int64	joins to the annotations
`cc3m_row`	int64	0-based row in `Train_GCC-training.tsv`
`url`	string	that row's image URL, the practical join key
`gap`	int32	rows skipped before this one when aligning, 0 for a clean match
`ambiguous`	bool	true where the alignment could not be pinned to one row

5,191 rows (0.17%) are flagged ambiguous; the remaining 99.83% resolve to a single CC3M row. A further 14,044 resolve to a row but carry an empty url, and are not flagged ambiguous, so joining on url reaches 3,002,596 samples (99.53%) and needs that filter separately. 285,709 rows have a nonzero gap, maximum 6.

Joining it to your CC3M

Join on url, not on position: every redistribution of CC3M is a different subset in a different order, because URLs rot at different rates for everyone. Expect roughly 90% of your copy to be covered, since GLIGEN grounded 3.0M of the 3.3M rows and your copy is itself a subset.

python

from huggingface_hub import snapshot_download

path = snapshot_download("freek23/cc3m-grounded-annotations", repo_type="dataset")

Boxes are in the coordinate frame given by width and height, which is the original image as GLIGEN received it. If your copy was resized to a square, the boxes must be padded or cropped with it or they will land in the wrong place.

Loader, join and geometry handling: https://github.com/fbyrman/vlm-geometry-ablation

Provenance

Derived from `gligen/cc3m_tsv` at revision 3f8d464e87669c5350787bdbc547f93dbb62357a, which packages CC3M images together with GLIGEN's grounding. The boxes and phrases are GLIGEN's; the contribution here is extracting them without the pixels and recovering which CC3M row each one belongs to.

That last part was necessary because GLIGEN's data_id indexes their own downloaded subset, not Train_GCC-training.tsv, so the annotations could not be joined onto any other CC3M copy as published. The two caption sequences are walked together to recover the mapping, and it was checked by re-downloading sampled URLs and confirming their dimensions agree with the recorded ones.

Licensing

The boxes and phrases come from GLIGEN, the captions and URLs from CC3M, and both carry their originators' terms. CC3M's terms govern the captions and URLs redistributed here; see the license link above. No image data is included.

Citation

Cite GLIGEN and Conceptual Captions, which produced the underlying annotations and dataset:

bibtex

@inproceedings{li2023gligen,
  title     = {{GLIGEN}: Open-Set Grounded Text-to-Image Generation},
  author    = {Li, Yuheng and Liu, Haotian and Wu, Qingyang and Mu, Fangzhou and
               Yang, Jianwei and Gao, Jianfeng and Li, Chunyuan and Lee, Yong Jae},
  booktitle = {CVPR},
  year      = {2023}
}

@inproceedings{sharma2018conceptual,
  title     = {Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset
               For Automatic Image Captioning},
  author    = {Sharma, Piyush and Ding, Nan and Goodman, Sebastian and Soricut, Radu},
  booktitle = {ACL},
  year      = {2018}
}