datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
newyorker_caption_contest
Dataset Card for New Yorker Caption Contest Benchmarks
Dataset Summary
See capcon.dev for more!
Data from:
Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest
@inproceedings{hessel2023androids,
title={Do Androids Laugh at Electric Sheep? {Humor} ``Understanding''
Benchmarks from {The New Yorker Caption Contest}},
author={Hessel, Jack and Marasovi{\'c}, Ana and Hwang, Jena D. and Lee, Lillian
and… See the full description on the dataset page: https://huggingface.co/datasets/jmhessel/newyorker_caption_contest.conceptual_captions
Dataset Card for Conceptual Captions
Dataset Summary
Conceptual Captions is a dataset consisting of ~3.3M images annotated with captions. In contrast with the curated style of other image caption annotations, Conceptual Caption images and their raw descriptions are harvested from the web, and therefore represent a wider variety of styles. More precisely, the raw descriptions are harvested from the Alt-text HTML attribute associated with web images. To arrive at the… See the full description on the dataset page: https://huggingface.co/datasets/google-research-datasets/conceptual_captions.pokemon-blip-captions
Notice of DMCA Takedown Action
We have received a DMCA takedown notice from The Pokémon Company International, Inc.
In response to this action, we have taken down the dataset.
We appreciate your understanding.
ramanv-image-captions-realCOCO-Caption
Large-scale Multi-modality Models Evaluation Suite
Accelerating the development of large-scale multi-modality models (LMMs) with lmms-eval
🏠 Homepage | 📚 Documentation | 🤗 Huggingface Datasets
This Dataset
This is a formatted version of COCO-Caption-2014-version. It is used in our lmms-eval pipeline to allow for one-click evaluations of large multi-modality models.
@misc{lin2015microsoft,
title={Microsoft COCO: Common Objects in Context}… See the full description on the dataset page: https://huggingface.co/datasets/lmms-lab-encoder/COCO-Caption.coco_captions
Dataset Card for "coco_captions"
More Information needed
youtube_caption_yue
YouTube ASR Caption Dataset (Cantonese)
This dataset was built from YouTube videos with manually provided captions in Cantonese. We used SenseVoice to re-transcribe the audio and filtered segments to build a high-quality collection of audio-caption pairs.
What’s included
Segments where the ASR output is identical to the original caption — likely clean.
Segments where differences are only homophones (同音字) or English words — likely ASR mistakes.
This combination supports… See the full description on the dataset page: https://huggingface.co/datasets/ming030890/youtube_caption_yue.ffhq512-captionCOCO-Caption2017
Large-scale Multi-modality Models Evaluation Suite
Accelerating the development of large-scale multi-modality models (LMMs) with lmms-eval
🏠 Homepage | 📚 Documentation | 🤗 Huggingface Datasets
This Dataset
This is a formatted version of COCO-Caption-2017-version. It is used in our lmms-eval pipeline to allow for one-click evaluations of large multi-modality models.
@misc{lin2015microsoft,
title={Microsoft COCO: Common Objects in Context}… See the full description on the dataset page: https://huggingface.co/datasets/lmms-lab-encoder/COCO-Caption2017.CaptionQA
📌 CaptionQA Benchmark
A high-density, taxonomy-grounded benchmark for evaluating image caption quality and the alignment between image information and generated captions
📄 Paper: CaptionQA: Is Your Caption as Useful as the Image Itself? 📦 Evaluation Code: GitHub Repository
Sample Usage
You can load the dataset using the Hugging Face datasets library:
from datasets import load_dataset
# Load the entire dataset
dataset = load_dataset("Borise/CaptionQA")
# Load a… See the full description on the dataset page: https://huggingface.co/datasets/Borise/CaptionQA.i1-captionsi1: A Simple and Fully Open Recipe for Strong Text-to-Image Models
Boya Zeng, Tianze Luo, Shu Pu, Jucheng Shen, Taiming Lu, Gabriel Sarch, Zhuang Liu
Princeton University
[arXiv][code][model][project page]
1. Overview
This dataset contains all captions used in our controlled experiments and the final training of the i1 model. Detailed instructions for downloading the corresponding images and matching the image-caption pairs can be found in our codebase.
2.… See the full description on the dataset page: https://huggingface.co/datasets/zlab-princeton/i1-captions.ramanv-image-captions-11Captioned_COCOStuffCOCO_captions_train
Dataset Card for "COCO_captions_train"
More Information needed
pokemon-gpt4-captions
Dataset Card for "pokemon-gpt4-captions"
This dataset is just lambdalabs/pokemon-blip-captions but the captions come from GPT-4 (Turbo).
Code used to generate the captions:
import base64
from io import BytesIO
import requests
from PIL import Image
def encode_image(image):
buffered = BytesIO()
image.save(buffered, format="JPEG")
img_str = base64.b64encode(buffered.getvalue())
returnimg_str.decode("utf-8")
def create_payload(image_string):
payload = {… See the full description on the dataset page: https://huggingface.co/datasets/diffusers/pokemon-gpt4-captions.GBC10M
Graph-based captioning (GBC) is a new image annotation paradigm that combines the strengths of long captions, region captions, and scene graphs
GBC interconnects region captions to create a unified description akin to a long caption, while also providing structural information similar to scene graphs.
** The associated data point can be found at demo/water_tower.json
Description and data format
The GBC10M dataset, derived from the original images in CC12M, is… See the full description on the dataset page: https://huggingface.co/datasets/graph-based-captions/GBC10M.220k-GPT4Vision-captions-from-LIVIS
220k-GPT4Vision-captions-from-LVIS
by: Christoph Schuhmann, Peter Bevan, 21 Nov, 2023
This dataset comprises 220,000 captioned images from the LVIS dataset. The captions were generated by summarising the LVIS-Instruct4V dataset released by X2FD. The instructions are converted into captions using Mistral-7B-OpenOrca.
PROMPT
"""<<SYS>> You are a highly intelligent, empathic, helpful, respectful, and honest assistant with high emotional intelligence. Always… See the full description on the dataset page: https://huggingface.co/datasets/laion/220k-GPT4Vision-captions-from-LIVIS.STAIR-Captions
Dataset Card for STAIR-Captions
Dataset Summary
STAIR Captions is a large-scale dataset containing 820,310 Japanese captions. This dataset can be used for caption generation, multimodal retrieval, and image generation.
Supported Tasks and Leaderboards
[More Information Needed]
Languages
The language data in JDocQA is in Japanese (BCP-47 ja-JP).
Dataset Structure
Data Instances
[More Information Needed]… See the full description on the dataset page: https://huggingface.co/datasets/shunk031/STAIR-Captions.nordjylland-news-image-captioning
Dataset Card for "nordjylland-news-image-captioning"
Dataset Summary
This dataset is a collection of image-caption pairs from the Danish newspaper TV2 Nord.
Supported Tasks and Leaderboards
Image captioning is the intended task for this dataset. No leaderboard is active at this point.
Languages
The dataset is available in Danish (da).
Dataset Structure
An example from the dataset looks as follows.
{
"file_name": "1.jpg",
"caption":… See the full description on the dataset page: https://huggingface.co/datasets/alexandrainst/nordjylland-news-image-captioning.COCO_captions_validation
Dataset Card for "COCO_captions_validation"
More Information needed
anime-captionsbokete_oogiri_captiondanbooru-multitier-captions-202606
Danbooru — multi-tier captions (202606)
Per-post Danbooru data for the 202606 crawl: native tags, the raw API metadata, model-generated
multi-tier natural-language captions (long / refined long / medium / short), and post flags.
One row per Danbooru post_id. Images are not included — each post is referenced by
post_id, danbooru_url, md5, and the Danbooru CDN URLs. (The two example previews below are
downscaled for illustration.)
Based on:… See the full description on the dataset page: https://huggingface.co/datasets/BootsofLagrangian/danbooru-multitier-captions-202606.wikiart_captionslearn_hf_food_not_food_image_captions
Food/Not Food Image Caption Dataset
Small dataset of synthetic food and not food image captions.
Text generated using Mistral Chat/Mixtral.
Can be used to train a text classifier on food/not_food image captions as a demo before scaling up to a larger dataset.
See Colab notebook on how dataset was created.
Example usage
import random
from datasets import load_dataset
# Load dataset
loaded_dataset = load_dataset("mrdbourke/learn_hf_food_not_food_image_captions")
# Get… See the full description on the dataset page: https://huggingface.co/datasets/mrdbourke/learn_hf_food_not_food_image_captions.movie-scenes-captionedmoss-character-voices-top3-captioned
MOSS Character Voices — Top-3 per Group, Captioned (training-ready)
The top-3 takes per group (rank 0/1/2 by reward model) of all ~12,900 groups in
laion/moss-character-voices-bestof64
— ~38,700 samples — each annotated with both LAION voice-acting caption pipelines and shipped with
all scores + metadata, ready to drop into training. Sharded as data/train-*-of-00129.parquet.
Captions (two pipelines, same clip)
caption_procedural — Procedural Voice Captions:
terse… See the full description on the dataset page: https://huggingface.co/datasets/laion/moss-character-voices-top3-captioned.laioncoco-captioned-sampleramanv-image-captions-11CONCEPTUAL_CAPTIONS_HU_FILTERED