datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
lotte_passagesLoTTE Passages Dataset for ColBERTv2msmarco_answerai_colbert_small_embeddings
MS MARCO ColBERT Embeddings
Pre-computed ColBERT embeddings for MS MARCO using PyLate and answerdotai/answerai-colbert-small-v1.
Dataset Structure
The dataset contains:
data/corpus/: 177 parquet files with document embeddings
data/queries/: 11 parquet files with query embeddings
data/qrels/train.parquet: Relevance judgments (532,751 pairs)
Usage
from datasets import load_dataset
# Load from directory (recommended for large datasets)
corpus =… See the full description on the dataset page: https://huggingface.co/datasets/WenxingZhu/msmarco_answerai_colbert_small_embeddings.lotteLoTTE Passages Dataset for ColBERTv2colbert-retrieval-mined-examplesmsmarco_token_score_colbertx_xlmr_large_zs_en_en
Dataset Card for "msmarco_token_score_colbertx_xlmr_large_zs_en_en"
More Information needed
lotte_pooled_colbertv2ms_marco_colbertv2ColBERT-TREC-COVIDThis dataset consists ColBERTv2.0 document vectors for the entire TREC-COVID dataset from BeIR. That 128 dimension per token, with 180 tokens for each of 171332 documents.
The dataset was created using A100-40GB sponsored by Qdrant. The code to create these vectors is here: https://colab.research.google.com/drive/1hEhyleSrBz_mPyQJnRc0MwBenDuX1ahY?usp=sharing
This dataset was created for indexing experiments by Qdrant.
xpr_colbertmiriad-mlateon-colbert-smoke
MIRIAD 200, encoded with mLateOn-medical
Multi-vector (ColBERT-style) embeddings for
tomaarsen/miriad-benchmark-200k,
produced with multi-vector-encoder/mLateOn-medical.
passages
200
token vectors
176,014
mean vectors / passage
880.07
dim
128
stored dtype
float16
embeddings size
0.05 GB
raw text encoded
1 MB
The embeddings are 49x larger than the text
they came from, which is why late-interaction retrieval needs quantization or pooling.… See the full description on the dataset page: https://huggingface.co/datasets/KShivendu/miriad-mlateon-colbert-smoke.ColBERT_Humor_Detection
ColBERT_Humor
Dataset Summary
ColBERT Humor contains 200,000 labeled short texts, equally distributed between humorous and non-humorous content. The dataset was created to overcome the limitations of prior humor detection datasets, which were characterized by inconsistencies in text length, word count, and formality, making them easy to predict with simple models without truly understanding the nuances of humor. The two sources for this dataset are the News Category… See the full description on the dataset page: https://huggingface.co/datasets/CreativeLang/ColBERT_Humor_Detection.msmarco-passage-train__tct_colbert__faiss2opq__M96_nbits8__ps159744__neg200__ibn__lr__
msmarco-passage-train__tct_colbert__faiss2opq__M96_nbits8__ps159744__neg200__ibn__lr__
Description
This is the PyTerrier JPQIndex for MSMARCO v1 passage corpus, which corresponds to an result from the SIGIR 2026 reproducibility paper.
Usage
# Load the artifact
import pyterrier as pt
import pyterrier_dr
index = pt.Artifact.from_hf('jpq-repro/msmarco-passage-train__tct_colbert__faiss2opq__M96_nbits8__ps159744__neg200__ibn__lr__')
model =… See the full description on the dataset page: https://huggingface.co/datasets/jpq-repro/msmarco-passage-train__tct_colbert__faiss2opq__M96_nbits8__ps159744__neg200__ibn__lr__.nq-train__tct_colbert__faiss2opq__M96_nbits8__ps159744__neg200__ibn__lr__
nq-train__tct_colbert__faiss2opq__M96_nbits8__ps159744__neg200__ibn__lr__
Description
This is the PyTerrier JPQIndex for the Wikipedia 2018 corpus used by Natural Questions (NQ).
Usage
# Load the artifact
import pyterrier as pt
import pyterrier_dr
index = pt.Artifact.from_hf('jpq-repro/nq-train__tct_colbert__faiss2opq__M96_nbits8__ps159744__neg200__ibn__lr__')
model = pyterrier_dr.TctColBert.hnp()
model.model =… See the full description on the dataset page: https://huggingface.co/datasets/jpq-repro/nq-train__tct_colbert__faiss2opq__M96_nbits8__ps159744__neg200__ibn__lr__.scidocs_colbert_pgtr_golden
SciDocs ColBERT PGTR Golden
SciDocs queries and corpus from Nithish2410/scidocs_colbert_pgtr_golden, with the previous targets ignored and replaced by full Qwen-reranked top-100 targets.
Contents
train.jsonl: 14,142 queries with 100 Qwen-reranked targets each.
items.jsonl: 25,657 SciDocs corpus passages.
Rerank Setup
Query source: existing query texts from the dataset.
Corpus source: existing items split from the dataset.
Candidate source:… See the full description on the dataset page: https://huggingface.co/datasets/Nithish2410/scidocs_colbert_pgtr_golden.msmarco_psg_v1.colbertv2
msmarco_psg_v1.colbertv2
Description
This is a ColBERT v2 index created via pyterrier_colbert2. This is the index that is used in the SIGIR 2026 PLAID-PRF paper.
Usage
# Download and load the index from HuggingFace
import pyterrier as pt
index = pt.Artifact.from_hf('pyterrier/msmarco_psg_v1.colbertv2',
plaid_mode=True, ncells=4,
centroid_score_threshold=0.4, ndocs=4096)
# TREC-DL 2019 pt.Experiment:
from pyterrier_colbert.ranking import _prf… See the full description on the dataset page: https://huggingface.co/datasets/pyterrier/msmarco_psg_v1.colbertv2.wikipedia_colbert_indexThis is an index of the English Wikipedia, made for WikiChat v1.0.
Refer to the following for more information:
GitHub repository: https://github.com/stanford-oval/WikiChat
Paper: WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia
WikiChat
Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia
Online demo:… See the full description on the dataset page: https://huggingface.co/datasets/stanford-oval/wikipedia_colbert_index.arxiv_colbert_pgtr_golden
ArXiv ColBERT PGTR Golden
ArXiv queries and corpus from Nithish2410/arxiv_colbert_pgtr_golden, with the previous targets ignored and replaced by Qwen-reranked top-100 targets.
Contents
train.jsonl: 10,000 queries with 100 Qwen-reranked targets each.
items.jsonl: 2,040 ArXiv corpus passages.
Rerank Setup
Query source: existing query texts from the dataset.
Corpus source: existing items split from the dataset.
Candidate source: e5-base-v2 retrieval… See the full description on the dataset page: https://huggingface.co/datasets/Nithish2410/arxiv_colbert_pgtr_golden.colbert_raw_datamsmarco-train-distil-colbert-v2msmarco-v2-colbertv2-fp32
MS MARCO v2 — ColBERTv2 fp32 Multi-Vector Embeddings (uncompressed)
Per-token multi-vector embeddings for the full MS MARCO v2 passage corpus
(~138.4M passages) plus the dev / dev2 queries, produced with the official
ColBERTv2 checkpoint (colbert-ir/colbertv2.0).
Precision: fp32 (NO residual quantization, NO pooling)
Dim: 128 per token
Corpus token vectors: ~9.41B (avg ~68 tokens/passage)
Total corpus size: ~4.82 TB
These embeddings are uncompressed on purpose (research on… See the full description on the dataset page: https://huggingface.co/datasets/yaooooo233/msmarco-v2-colbertv2-fp32.msmarco-2m-triplets-colbert-formatpgturbohybrid_dbpedia_colbert
johannhartmann/pgturbohybrid_dbpedia_colbert
Precomputed DBpedia ColBERT multivectors for pgturbohybrid benchmark runs.
The dataset stores packed little-endian float16 values for the document and query embeddings.
Importing these rows into PostgreSQL avoids llama.cpp embedding generation
during retrieval/index benchmarks.
This compact export stores half-precision values and reconstructs turbohybrid_multivector values on import. Use it for benchmark loading where avoiding runtime… See the full description on the dataset page: https://huggingface.co/datasets/johannhartmann/pgturbohybrid_dbpedia_colbert.finetuning-colbert-acr-syntheticcovid_colbert_pgtr_golden
COVID ColBERT PGTR Golden
COVID queries and corpus from Nithish2410/covid_colbert_pgtr_golden, with the previous targets ignored and replaced by Qwen-reranked top-100 targets.
Contents
train.jsonl: 10,000 queries with 100 Qwen-reranked targets each.
items.jsonl: 171,332 COVID corpus passages.
Rerank Setup
Query source: existing query texts from the dataset.
Corpus source: existing items split from the dataset.
Candidate source: e5-base-v2… See the full description on the dataset page: https://huggingface.co/datasets/Nithish2410/covid_colbert_pgtr_golden.bright.sustainable.colbert2
bright.sustainable.colbert2
Description
An index of the sustainable-living subset of the BRIGHT dataset.
Usage
You'll need http://github.com/cmacdonald/pyterrier_colbert2
# Load the artifact
import pyterrier as pt
index = pt.Artifact.from_hf('pyterrier-tutorial/bright.sustainable.colbert2')
from pyterrier.measures import *
topics = pt.get_dataset('irds:bright/sustainable-living').get_topics(variant='text')
qrels =… See the full description on the dataset page: https://huggingface.co/datasets/pyterrier-tutorial/bright.sustainable.colbert2.commonsense_qa_retrieved_by_colbert
Dataset Card for "commonsense_qa_retrieved_by_colbert"
This is the validation set of CSQA, with each question retrieved from ColBERT v2 trained on MS MARCO Passage Ranking (https://downloads.cs.stanford.edu/nlp/data/colbert/colbertv2/colbertv2.0.tar.gz).
We index the question part of the train set using doc_maxlen=30, nbits=2. We search each question of validation set with k=10 and put the results in the retrieved column.
ODQA_colbert_top5_100wordsmsmarco_colbertcolbert-wiki2017colbert-wiki17-assets