Panorama-grounding/PanoCaps

PanoCaps: A Human-Annotated Benchmark for Panoptic Grounded Captioning

   

<p align="center"> <img src="https://huggingface.co/datasets/Panorama-grounding/PanoCaps/resolve/main/panocaps_card.png" width="100%" alt="PanoCaps example: grounded masks, grounded phrases and the grounded caption" /> </p>

PanoCaps is a benchmark for panoptic grounded captioning: a model writes a full-scene caption and grounds every mentioned entity, things and stuff alike, to pixel-level masks.

It contains 3,470 images and 34K panoptic regions, averaging ~9 grounded entities per image, with >99% of regions grounded. Captions are human-written and verified, cover the entire visible scene, use open-vocabulary descriptions that go beyond category labels, and support one-to-many and many-to-one text ↔ mask mappings. Masks follow COCO-style panoptic annotations.

For research and non-commercial use only.

See also our cleaned version of the COCONut-PanCap training captions, COCONut-PanCap Re-captioned.

Images

Images are not included in this repository. Download them from the source datasets below, or get both images and annotations, already formatted, as a single archive from this Google Drive link. The annotations reference the images by file_name and id.

Download

# annotations only (load_dataset reads the parquet in data/, not these)
hf download Panorama-grounding/PanoCaps --repo-type dataset --include "annotations/*" \
    --local-dir /path/to/data/PanoCaps

# or images + annotations, pre-formatted, from the Google Drive archive above
pip install gdown && gdown 1iNEuKWJdv1wKKGycP-4hTKHdfmcuQEOB -O PanoCaps.zip
unzip PanoCaps.zip -d /path/to/data/

Formats

The same annotations are published in two formats:

• —`annotations/` — COCO-style JSON, as <split>_caption.json / <split>_mask.json pairs for train, val and test. Recommended for training and evaluation. Join the two files on image_id / id.
• —`data/` — parquet, merging caption and mask information into one entry per image. Recommended for browsing in the Dataset Viewer.

Note that load_dataset reads only data/; the JSON annotations have to be downloaded explicitly.

Caption files

annotations[*]:

`mask_ids` are positional indices into that image's list of masks — equivalently, into labels — not the global id values used in the mask file. The same mask may be referenced by several spans, for instance when an entity is mentioned again later in the caption.

Mask files

annotations[*]:

Both caption and mask files also carry images[*] with file_name, height, width, id and data_source.

Annotation

Images were curated from ADE20K, COCONut and VIPSeg for visual quality and suitability for dense grounding. Paid professional annotators then wrote long-form scene descriptions covering attributes, relationships and all visible entities, tagged every textual reference with <mask_ids:description> markers, and produced the corresponding label_matched mapping. A separate quality-control pass verified grounding ids, completeness of region coverage and annotation consistency.

The base panoptic masks come from the source datasets; all captions and grounding annotations were created specifically for PanoCaps.

License

This repository merges, normalizes and redistributes content from existing datasets, and is therefore released strictly for research and non-commercial use. Commercial use is not permitted, and users must comply with the licenses of each source dataset.

Citation

If you find our work useful for your research, please consider citing our paper:

@article{pieri2026panorama,
  title   = {PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection},
  author  = {Pieri, Sara and Kazakos, Evangelos and Chen, Shizhe and Sivic, Josef and Schmid, Cordelia},
  journal = {arXiv preprint arXiv:2609.19143},
  year    = {2026}
}