BreakpointAI/test-dataset-8

Breakpoint Grounding 55M

Quick start

from datasets import load_dataset
ds = load_dataset("BreakpointAI/breakpoint-grounding-55m", split="train")
ds[0]  # {'image': <PIL.Image>, 'image_caption': ..., 'object_captions': [...], 'normalized_boxes': [...], 'img_size_wh': [...]}

Dataset summary

Breakpoint Grounding 55M is, to our knowledge, the largest instance-grounded image–text dataset released publicly. Every image comes with an image-level caption, a set of object bounding boxes, and a caption for every box. It was built by Breakpoint AI to train a 10B-parameter joint image + bounding-box diffusion model for synthetic object-detection data generation.

<!-- Once the example figure is ready, uncomment and upload it (see updatehubcard.py --asset): [image] -->

• —54,909,477 images, each with one image-level caption
• —511,652,664 bounding boxes (9.32 per image on average), each with its own caption

For comparison, ROVI (ICCV 2025), the most recent comparable instance-grounded dataset, labels 1M web images with a similar VLM + open-vocabulary-detector pipeline — this dataset is more than 50× larger.

Why this dataset exists

At Breakpoint, we trained diffusion models to generate synthetic training data for object detection models. Our ultimate model was a 10-billion-parameter joint diffusion model that generated images and bounding boxes simultaneously. Given 10 labeled images, it could adapt to a new scene and start producing labeled training images.

In academia, there has been considerable research on conditional generation (bounding boxes to image) <!-- TODO: cite GLIGEN, InstanceDiffusion, etc. -->, and the associated datasets <!-- TODO: cite -->. While helpful, this work did not touch joint generation, and unfortunately was not at the scale necessary to be actually useful.

In order to get a better performing model, we decided to build this dataset, which is over 50 times larger than comparable datasets (e.g. ROVI, ICCV 2025, at 1M images). Having this much data is what allowed us to train a large and high-performing model.

Image sources

Images are drawn from four public web-image collections:

How it was built

• —Images: collected from the four sources above, resized to ~1MP, light integrity filtering (corrupt / unreadable files removed).
• —Annotations (image_caption, object_captions, normalized_boxes): generated by Breakpoint AI's automated annotation pipeline (object detection + captioning models). They are not human-labeled. See Appendix: Annotation pipeline for details.

Dataset structure

Fields

Bounding box format

Each box is [x_min, y_min, x_max, y_max] (xyxy), with coordinates normalized to [0, 1] relative to image width/height. normalized_boxes[i] corresponds to object_captions[i].

Splits

Example

from datasets import load_dataset

ds = load_dataset("BreakpointAI/breakpoint-grounding-55m", split="train", streaming=True)
row = next(iter(ds))

row["image"]              # PIL.Image
row["image_caption"]      # "a photograph of ..."
row["object_captions"]    # ["a dog", "a red ball", ...]
row["normalized_boxes"]   # [[0.12, 0.34, 0.56, 0.78], ...]
row["img_size_wh"]        # [1820, 1024]

Licensing and responsible use

Released for research and educational use only.

This dataset aggregates images from multiple public sources, each under its own terms (see Image sources above). The bounding boxes and captions contributed by Breakpoint AI are released under CC BY 4.0. Use of the images is additionally governed by the licenses and terms of their original sources; downstream users are responsible for complying with those, including any attribution requirements (notably for Wikimedia Commons and Open Images) and the RedCaps non-commercial research terms.

Personal data. The images depict real, sometimes identifiable, people and places. Material being publicly posted does not remove it from the scope of data-protection laws such as the GDPR, UK GDPR, and CCPA. Do not use this dataset to identify, profile, track, surveil, or contact individuals, or to train biometric identification systems.

No affiliation. This dataset is not affiliated with, endorsed by, or connected to Google / Open Images, the Wikimedia Foundation, Reddit, or LAION e.V.

Removal requests. If you are the rights holder for an image, or are depicted in one, and want it removed, open a discussion on this repository identifying the affected row indices (or attaching the image) and the affected rows will be removed.

Limitations and biases

• —Annotations are model-generated and contain errors: missed objects, spurious boxes, imprecise coordinates, and caption hallucinations.
• —Inherits the content distribution and biases of the four source collections, including web-scale skews in geography, language, subject matter, and the demographics of who posts images to Reddit and Wikimedia.
• —Images vary in aesthetic and resolution quality; only a light integrity filter was applied.

Acknowledgements

• —Hosted on the Hugging Face Hub with a public dataset storage grant.
• —Thanks to Daniel van Strien and the Hugging Face datasets team.
• —Images from Open Images, Wikimedia Commons, RedCaps, and LAION.

Appendix: Annotation pipeline

<!-- TODO: fill in — which object detector (Jamie mentioned Grounding DINO), which captioning model(s), and the prompts used for each. Franklin has this detail. -->

Citation

@misc{breakpoint_grounding_55m,
  title  = {Breakpoint Grounding 55M},
  author = {Wang, Franklin and {[Desmond LASTNAME]} and Murdoch, Jamie},
  year   = {2026},
  url    = {https://huggingface.co/datasets/BreakpointAI/breakpoint-grounding-55m}
}

<!-- TODO: confirm Desmond's last name and Jamie's preferred citation name (Jamie Murdoch vs. William Murdoch), and author order (currently Franklin, Desmond, Jamie per Slack). -->

Related work

@inproceedings{peng2025rovi,
  title     = {ROVI: A VLM-LLM Re-Captioned Dataset for Open-Vocabulary
               Instance-Grounded Text-to-Image Generation},
  author    = {Peng, Cihang and Hou, Qiming and Ren, Zhong and Zhou, Kun},
  booktitle = {ICCV},
  year      = {2025}
}

@inproceedings{li2023gligen,
  title     = {GLIGEN: Open-Set Grounded Text-to-Image Generation},
  author    = {Li, Yuheng and Liu, Haotian and Wu, Qingyang and Mu, Fangzhou and Yang, Jianwei
               and Gao, Jianfeng and Li, Chunyuan and Lee, Yong Jae},
  booktitle = {CVPR},
  year      = {2023}
}

@inproceedings{wang2024instancediffusion,
  title     = {InstanceDiffusion: Instance-level Control for Image Generation},
  author    = {Wang, Xudong and Darrell, Trevor and Rambhatla, Sai Saketh and Girdhar, Rohit
               and Misra, Ishan},
  booktitle = {CVPR},
  year      = {2024}
}

<!-- TODO: verify these two entries (author order / exact venue) before publishing — drafted from memory, not looked up. -->