CoolFace
Datasetpublic

hmdliu/imagenet-captions-florence2l

ImageNet-1k captions (Florence-2-large) + CLIP ViT-L/14 text embeddings One caption per ImageNet-1k image (train 1,281,167 + val 50,000) and its CLIP text embedding, made for caption-conditioned latent-diffusion experiments (pcls-exp wave 5). No images are included. How they were made Captioner: florence-community/Florence-2-large (777M, native transformers>=4.57), task <DETAILED_CAPTION>, greedy decoding, max_new_tokens=80, fp16. Input crop: the centre square of… See the full description on the dataset page: https://huggingface.co/datasets/hmdliu/imagenet-captions-florence2l.

sourceHugging Faceotherupdated 10d agoView on Hugging Face
0likes76downloads
Dataset Card

ImageNet-1k captions (Florence-2-large) + CLIP ViT-L/14 text embeddings

One caption per ImageNet-1k image (train 1,281,167 + val 50,000) and its CLIP text embedding, made for caption-conditioned latent-diffusion experiments (pcls-exp wave 5). No images are included.

How they were made

  • —Captioner: florence-community/Florence-2-large (777M, native transformers>=4.57), task <DETAILED_CAPTION>, greedy decoding, max_new_tokens=80, fp16.
  • —Input crop: the centre square of the image (shortest side, centred) resized to Florence's 768x768 — the same content as Resize(256) + CenterCrop(256) used by the latent pipeline.
  • —Text embedding: openai/clip-vit-large-patch14 text tower, CLIPTextModelWithProjection(...).text_embeds — 768-d, not L2-normalised (norm 11.6 +- 0.4), 77-token truncation (0.28 % of captions truncated). Before encoding, the leading template "The image shows " (regex ^(the|this) (image|picture|photo) (shows|is of|depicts|features) ) is stripped; the raw caption in the jsonl keeps it.

Files

filecontent
txt_train.npyfloat16 [1281167, 768], row i = embedding of train image i in ImageFolder order (see below)
txt_val.npyfloat16 [50000, 768], same for the val split
train_shard{00..15}of16.jsonl, val_shard{00,01}of02.jsonlone line per image: {"index", "label", "path", "caption"}; index is the ImageFolder position, caption is the raw Florence output (prefix included), path is the source machine's path (basename is what matters)
imagefolder_train.json, imagefolder_val.jsonthe exact ordering: list of [path, label] in index order
txt_stats.json, probe_val50k.jsonembedding statistics and a cross-modal probe (below)
samples_val_12.png12 val image/caption pairs

ImageFolder order = torchvision.datasets.ImageFolder on the standard ILSVRC2012 layout (<root>/<synset>/<file>): classes sorted by synset id, files sorted by name inside each class (train: n01440764_10026.JPEG first; val: ILSVRC2012_val_00000293.JPEG first, i.e. the val images are in their synset folders). Re-encoded copies with a different extension keep the same order; verify with the basenames in imagefolder_*.json.

Sanity numbers (val 50k, CLIP ViT-L/14 image tower vs these text embeddings)

cos(text, own image) 0.288, same-class image 0.212, random image 0.067; the own image is the top-1 of 50k for 36 % of captions, median rank 3. Text 1-NN class purity 0.63, image 1-NN 0.71.

Loading

python
import numpy as np, json
T = np.load("txt_train.npy").astype(np.float32)          # [1281167, 768]
order = json.load(open("imagefolder_train.json"))       # [[path, label], ...] same order