datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
FineWeb-10B
Qdrant-FineWeb-10B
Overview
Qdrant-FineWeb-10B (Q-FineWeb-10B) is a 10-billion-vector retrieval benchmark derived from FineWeb. Each document is represented with dense and sparse embeddings from Alibaba-NLP/gte-multilingual-base, alongside its original FineWeb payload and metadata. The benchmark also includes exact brute-force ground truth for ~120,000 MS MARCO queries.
The dataset includes:
10 billion dense embeddings
10 billion sparse embeddings
FineWeb… See the full description on the dataset page: https://huggingface.co/datasets/Qdrant/FineWeb-10B.gaianet-qdrant-snapshotPubMed-MV
PubMed-MultiVector (PubMed-VE)
Overview
PubMed-VE is a retrieval benchmark built from PubMed abstracts using BGE-M3.
Each document is represented in all three formats produced by BGE-M3:
~24 million dense embeddings
~24 million sparse embeddings
~8.77B multi-vector (token-level) embeddings
Statistic
Value
Documents
23,898,701
Queries
1,000
Ground truth
exact top-1000, one list per modality (dense / sparse / multi-vector)
Embeddings
BAAI/bge-m3… See the full description on the dataset page: https://huggingface.co/datasets/Qdrant/PubMed-MV.wikipedia_20240401_10-languages_bge-m3_qdrant_indexThis repository contains a Qdrant index created from preprocessed and chunked Wikipedia HTML dumps from 10 languages. The embedding model used is BAAI/bge-m3
This index is compatible with WikiChat v2.0.
Refer to the following for more information:
GitHub repository: https://github.com/stanford-oval/WikiChat
Papers:
WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia
SPAGHETTI: Open-Domain Question Answering from Heterogeneous Data Sources… See the full description on the dataset page: https://huggingface.co/datasets/stanford-oval/wikipedia_20240401_10-languages_bge-m3_qdrant_index.Coyo-VE
Coyo-Vector-Embeddings (Coyo-VE)
Overview
Coyo-VE is a large-scale visual-text-embedding slice of the COYO subset from the LLaVA-OneVision-1.5-Mid-Training-85M dataset, with the original images being sourced from coyo-700m. Each image-caption pair is jointly embedded using Qwen3-VL-Embedding-2B, which accepts mixed image-and-text inputs and encodes their combined visual and textual content into a single 2,048-dimensional dense vector. This produces a unified… See the full description on the dataset page: https://huggingface.co/datasets/Qdrant/Coyo-VE.arxiv-titles-instructorxl-embeddings
arxiv-titles-instructorxl-embeddings
This dataset contains 768-dimensional embeddings generated from the arxiv
paper titles using InstructorXL model. Each
vector has an abstract used to create it, along with the DOI (Digital Object Identifier). The
dataset was created using precomputed embeddings exposed by the Alexandria Index.
Generation process
The embeddings have been generated using the following instruction:
Represent the Research Paper title for retrieval;… See the full description on the dataset page: https://huggingface.co/datasets/Qdrant/arxiv-titles-instructorxl-embeddings.dbpedia-entities-openai3-text-embedding-3-large-1536-1M1M OpenAI Embeddings: text-embedding-3-large 1536 dimensions
Created: February 2024.
Text used for Embedding: title (string) + text (string)
Embedding Model: OpenAI text-embedding-3-large
This dataset was generated from the first 1M entries of https://huggingface.co/datasets/BeIR/dbpedia-entity, extracted by @KShivendu_ here
dbpedia-entities-openai3-text-embedding-3-large-3072-1M1M OpenAI Embeddings: text-embedding-3-large 3072 dimensions + ada-002 1536 dimensions — parallel dataset
Created: February 2024.
Text used for Embedding: title (string) + text (string)
Embedding Model: text-embedding-3-large
This dataset was generated from the first 1M entries of https://huggingface.co/datasets/BeIR/dbpedia-entity, extracted by @KShivendu_ here
wikipedia_20240801_10-languages_bge-m3_qdrant_indexThis repository contains a Qdrant index created from preprocessed and chunked Wikipedia HTML dumps from 10 languages. The embedding model used is BAAI/bge-m3
This index is compatible with WikiChat v2.0.
Refer to the following for more information:
GitHub repository: https://github.com/stanford-oval/WikiChat
Papers:
WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia
SPAGHETTI: Open-Domain Question Answering from Heterogeneous Data Sources… See the full description on the dataset page: https://huggingface.co/datasets/stanford-oval/wikipedia_20240801_10-languages_bge-m3_qdrant_index.arxiv_20240801_gte-base-en-v1.5_qdrant_indexThis repository contains a Qdrant index created from preprocessed and chunked arxiv papers from Semantic Scholar. The embedding model used is Alibaba-NLP/gte-large-en-v1.5.
This index is compatible with WikiChat v2.0.
Refer to the following for more information:
GitHub repository: https://github.com/stanford-oval/WikiChat
Papers:
WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia
SPAGHETTI: Open-Domain Question Answering from Heterogeneous… See the full description on the dataset page: https://huggingface.co/datasets/stanford-oval/arxiv_20240801_gte-base-en-v1.5_qdrant_index.en_qdrant_wikipedia
Wikipedia English Embeddings Dataset (Qdrant Snapshot)
This dataset contains a complete Qdrant vector database snapshot of Wikipedia English articles, each chunked and embedded with both SPLADE and BM25 sparse embeddings, yielding a total of 130M embeddings. Chunks are linked to their neighbors and the main text to make full context retrieval simple and fast. If you'd rather access a hosted version of this (with the latest Wiki updates indexed automatically), head over to the… See the full description on the dataset page: https://huggingface.co/datasets/EmergentMethods/en_qdrant_wikipedia.arxiv-abstracts-instructorxl-embeddings
arxiv-abstracts-instructorxl-embeddings
This dataset contains 768-dimensional embeddings generated from the arxiv
paper abstracts using InstructorXL model. Each
vector has an abstract used to create it, along with the DOI (Digital Object Identifier). The
dataset was created using precomputed embeddings exposed by the Alexandria Index.
Generation process
The embeddings have been generated using the following instruction:
Represent the Research Paper abstract for… See the full description on the dataset page: https://huggingface.co/datasets/Qdrant/arxiv-abstracts-instructorxl-embeddings.wolt-food-clip-ViT-B-32-embeddings
wolt-food-clip-ViT-B-32-embeddings
Qdrant's Food Discovery demo relies on the dataset of food images from the Wolt
app. Each point in the collection represents a dish with a single image. The image is represented as a vector of 512
float numbers.
Generation process
The embeddings generated with clip-ViT-B-32 model have been generated using the following code snippet:
from PIL import Image
from sentence_transformers import SentenceTransformer
image_path =… See the full description on the dataset page: https://huggingface.co/datasets/Qdrant/wolt-food-clip-ViT-B-32-embeddings.hm_ecommerce_products
H&M Personalized Fashion Recommendations - Enhanced Dataset
Dataset Description
This dataset is a processed and enhanced version of the H&M Personalized Fashion Recommendations Kaggle competition dataset. The original dataset has been cleaned and augmented with pre-computed embeddings and accessible image URLs to facilitate fashion recommendation research and multimodal retrieval applications.
Dataset Summary
The H&M dataset contains rich product metadata… See the full description on the dataset page: https://huggingface.co/datasets/Qdrant/hm_ecommerce_products.NOAA-Buoy
NOAA Buoy meterological data
NOAA Buoy Data was downloaded, processed, and cleaned for tasks pertaining to tabular data. The data consists of meteorological measurements. There are two datasets
From 1980 through 2022 (denoted with "years" in file names)
From Jan 2023 through end of Sept 2023 (denoted with "2023" in file names)
The original intended use is for anomaly detection in tabular data.
Dataset Details
Dataset Description
This dataset contains weather… See the full description on the dataset page: https://huggingface.co/datasets/Qdrant/NOAA-Buoy.dbpedia-entities-openai3-text-embedding-3-small-1536-100Kdbpedia-entities-openai3-text-embedding-3-large-1536-100KBGE-m3-1-million-adsdbpedia-entities-openai3-text-embedding-3-large-3072-100Kqdrant_docColBERT-TREC-COVIDThis dataset consists ColBERTv2.0 document vectors for the entire TREC-COVID dataset from BeIR. That 128 dimension per token, with 180 tokens for each of 171332 documents.
The dataset was created using A100-40GB sponsored by Qdrant. The code to create these vectors is here: https://colab.research.google.com/drive/1hEhyleSrBz_mPyQJnRc0MwBenDuX1ahY?usp=sharing
This dataset was created for indexing experiments by Qdrant.
google-landmark-geo
Dataset Card for Geo Coordinate Augmented Google-Landmarks
Geo coordinates were added as data to a tar file's worth of images from the Google Landmark V2. Not all of the
images could be geo-tagged due to lack of coordinates on the image's wikimedia page.
Dataset Details
Dataset Description
Geo coordinates were added as data to a tar file's worth of images from the Google Landmark V2. There were many more images that could have
been downloaded but this dataset… See the full description on the dataset page: https://huggingface.co/datasets/Qdrant/google-landmark-geo.dbpedia-entities-openai3-text-embedding-3-small-512-100Kwiki-1m-qdrant-snapshot
Wikipedia 1M — Embedding Snapshot (Qdrant, 768D, GTE Multilingual Base)
This dataset contains a 7GB Qdrant snapshot with 1,000,000 Polish Wikipedia passages, embedded using:
Model: Alibaba-NLP/gte-multilingual-base
Embedding dimension: 768
Distance metric: cosine
Index type: HNSW (M=32, ef_construct=256, on-disk enabled)
Chunking strategy: semantic, max chunk size 512, overlap 128
Payloads: include passage text + metadata
The snapshot can be restored directly using the… See the full description on the dataset page: https://huggingface.co/datasets/Floressek/wiki-1m-qdrant-snapshot.qdrant_doc_qnadbpedia-entities-openai3-text-embedding-3-small-1024-100Kdbpedia-entities-openai3-text-embedding-3-large-1024-100Kcds-qdrant-snapshotsqdrant-landing-page-2024-06-21material-Qdrant-db