Datasetpublic

scaleinvariant/paired-open-images-embedded-pe-core-g14-448

Paired Open Images with PE-Core-G14-448 Embeddings This dataset contains pairs of images from Open Images along with their embeddings computed using Meta's Perception Encoder (PE-Core-G14-448). Each row contains two images (as JPEG bytes), their metadata, and their corresponding 1280-dimensional embeddings. Data Layout Column Description image1_jpeg JPEG bytes for the first image image1_metadata Metadata for the first image image2_jpeg JPEG bytes… See the full description on the dataset page: https://huggingface.co/datasets/scaleinvariant/paired-open-images-embedded-pe-core-g14-448.

sourceHugging Facecc-by-4.0updated 7mo agoView on Hugging Face

0likes336downloads

Dataset Card

Paired Open Images with PE-Core-G14-448 Embeddings

This dataset contains pairs of images from Open Images along with their embeddings computed using Meta's Perception Encoder (PE-Core-G14-448).

Each row contains two images (as JPEG bytes), their metadata, and their corresponding 1280-dimensional embeddings.

Data Layout

Column	Description
`image1_jpeg`	JPEG bytes for the first image
`image1_metadata`	Metadata for the first image
`image2_jpeg`	JPEG bytes for the second image
`image2_metadata`	Metadata for the second image
`image1_embedding0`	PE-Core-G14-448 embedding (dim=1280) for image 1
`image2_embedding0`	PE-Core-G14-448 embedding (dim=1280) for image 2

Splits

Split	Files
train	94 parquet shards
validation	20 parquet shards
test	5 parquet shards

Using the Embeddings

The embeddings behave like CLIP embeddings. You can use them for zero-shot classification, retrieval, or similarity search.

Generating text embeddings for comparison

bash

git clone https://github.com/facebookresearch/perception_models.git
cd perception_models

python

import torch
import core.vision_encoder.pe as pe
import core.vision_encoder.transforms as transforms

model = pe.CLIP.from_config("PE-Core-G14-448", pretrained=True).cuda().eval()
tokenizer = transforms.get_text_tokenizer(model.context_length)

text_tokens = tokenizer(["dog", "cat"]).cuda()
with torch.no_grad(), torch.autocast("cuda"):
    _, text_features, _ = model(None, text_tokens)

# Compare with stored embeddings via cosine similarity
image_features = ...  # load from parquet
image_features = image_features / image_features.norm(dim=-1, keepdim=True)
text_features = text_features / text_features.norm(dim=-1, keepdim=True)
similarity = model.logit_scale.exp() * image_features @ text_features.T
probs = similarity.softmax(dim=-1)

License

The images originate from Open Images, which is licensed under CC BY 4.0.