Qdrant/Coyo-VE

Coyo-Vector-Embeddings (Coyo-VE)

Overview

Coyo-VE is a large-scale visual-text-embedding slice of the COYO subset from the LLaVA-OneVision-1.5-Mid-Training-85M dataset, with the original images being sourced from coyo-700m. Each image-caption pair is jointly embedded using Qwen3-VL-Embedding-2B, which accepts mixed image-and-text inputs and encodes their combined visual and textual content into a single 2,048-dimensional dense vector. This produces a unified multimodal representation for each data point, enabling retrieval based jointly on its visual and textual semantics.

The benchmark includes 25,000 joint image-text queries selected from the ImageNet subset of LLaVA-OneVision-1.5-Mid-Training-85M. These queries model a social-media retrieval scenario in which a user searches for posts related to a given image-text query.

Headline statistics:

Dataset Contents

The corpus is derived from the COYO subset of LLaVA-OneVision-1.5-Mid-Training-85M, which is itself derived from COYO-700M. Embeddings and ground truth were produced with the nova-embed and nova-bf components of the Supernova pipeline

Each corpus item contains a paired image and textual caption.

Each row contains the document/image payload along with its high-dimensional vector embedding:

Embeddings

Each image-caption pair is jointly embedded using Qwen3-VL-Embedding-2B.

• —Modality: joint image + text (input_column: text=caption,image=image, modality: multimodal)
• —Representation: dense
• —Dimensions: 2,048
• —Datatype: float32
• —Similarity: cosine

Queries and Ground Truth

The benchmark includes 25,000 joint image-text queries and ground truth results at k=1000, together in queries/queries_gt_k1000.parquet. Each query consists of an image and associated text that are jointly embedded using the same Qwen3-VL-Embedding-2B representation as the corpus. The queries were selected from the ImageNet subset of LLaVA-OneVision-1.5-Mid-Training-85M, and include the following fields.

Ground truth is exact: every hit list holds 1,000 unique corpus ids in strictly descending score order, and hit_ids joins directly against the corpus id column.

Downloading and Joining the Images

This release contains embeddings and captions but no image bytes. The images live upstream in `mvp-lab/LLaVA-OneVision-1.5-Mid-Training-85M`, keyed by the same id this release uses, and join_image_urls.py fetches them and joins them to the rows for you:

# Try it on 1,000 rows first.
./join_image_urls.py --limit 1000

# The whole corpus (15.4M rows, several TB read from upstream).
./join_image_urls.py

# The benchmark queries instead of the corpus.
./join_image_urls.py --what queries --limit 100

The script needs only pyarrow and huggingface_hub.

It writes the image files into ./images/ and a Parquet — corpus_with_images.parquet, or queries_with_images.parquet for queries — carrying the release's own columns plus image_path:

Useful flags: --images DIR to write elsewhere, --limit N to stop early, --shards to point at part of the corpus (say 'hf://datasets/Qdrant/Coyo-VE/data/000/*.parquet') or at a local copy.

Only what you ask for is downloaded. Each upstream file's footer statistics rule the file out without reading a single column, so the id column is read only for the surviving row groups and the image bytes only where a wanted id actually sits — fetching two images touched columns in 5 of 63 upstream shards.

Intended Uses

Coyo-VE is intended for evaluating multimodal retrieval systems in workloads where images and text jointly determine semantic similarity.

Potential uses include:

• —multimodal vector search
• —image-text retrieval
• —approximate nearest-neighbor search
• —vector database ingestion and indexing
• —distributed retrieval systems
• —retrieval over social-media-style image-text posts
• —large-scale multimodal embedding evaluation

Licensing

This dataset slice and its metadata are made available under the CC-BY-4.0 license, following the upstream license terms of the COYO dataset and LLaVA-OneVision-1.5 release.

Acknowledgments

We thank HuggingFace for providing a grant to offset a portion of the cost of hosting this dataset. Additionally, Coyo-VE would not exist without the upstream work it builds on:

• —the MVP-Lab / LLaVA-OneVision-1.5 authors, for assembling and recaptioning the 85M mid-training corpus;
• —Kakao Brain, for collecting and releasing COYO-700M;
• —the Qwen team, for releasing Qwen3-VL-Embedding-2;
• —the ImageNet authors, whose collection supplies the benchmark query items;
• —the vLLM project, which serves the embedding model in this pipeline.