datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
msmarco_answerai_colbert_small_embeddings
MS MARCO ColBERT Embeddings
Pre-computed ColBERT embeddings for MS MARCO using PyLate and answerdotai/answerai-colbert-small-v1.
Dataset Structure
The dataset contains:
data/corpus/: 177 parquet files with document embeddings
data/queries/: 11 parquet files with query embeddings
data/qrels/train.parquet: Relevance judgments (532,751 pairs)
Usage
from datasets import load_dataset
# Load from directory (recommended for large datasets)
corpus =… See the full description on the dataset page: https://huggingface.co/datasets/WenxingZhu/msmarco_answerai_colbert_small_embeddings.msmarco_token_score_colbertx_xlmr_large_zs_en_en
Dataset Card for "msmarco_token_score_colbertx_xlmr_large_zs_en_en"
More Information needed
xpr_colbertmiriad-mlateon-colbert-smoke
MIRIAD 200, encoded with mLateOn-medical
Multi-vector (ColBERT-style) embeddings for
tomaarsen/miriad-benchmark-200k,
produced with multi-vector-encoder/mLateOn-medical.
passages
200
token vectors
176,014
mean vectors / passage
880.07
dim
128
stored dtype
float16
embeddings size
0.05 GB
raw text encoded
1 MB
The embeddings are 49x larger than the text
they came from, which is why late-interaction retrieval needs quantization or pooling.… See the full description on the dataset page: https://huggingface.co/datasets/KShivendu/miriad-mlateon-colbert-smoke.ColBERT-TREC-COVIDThis dataset consists ColBERTv2.0 document vectors for the entire TREC-COVID dataset from BeIR. That 128 dimension per token, with 180 tokens for each of 171332 documents.
The dataset was created using A100-40GB sponsored by Qdrant. The code to create these vectors is here: https://colab.research.google.com/drive/1hEhyleSrBz_mPyQJnRc0MwBenDuX1ahY?usp=sharing
This dataset was created for indexing experiments by Qdrant.
msmarco-train-distil-colbert-v2ODQA_colbert_top5_100wordspgturbohybrid_dbpedia_colbert
johannhartmann/pgturbohybrid_dbpedia_colbert
Precomputed DBpedia ColBERT multivectors for pgturbohybrid benchmark runs.
The dataset stores packed little-endian float16 values for the document and query embeddings.
Importing these rows into PostgreSQL avoids llama.cpp embedding generation
during retrieval/index benchmarks.
This compact export stores half-precision values and reconstructs turbohybrid_multivector values on import. Use it for benchmark loading where avoiding runtime… See the full description on the dataset page: https://huggingface.co/datasets/johannhartmann/pgturbohybrid_dbpedia_colbert.commonsense_qa_retrieved_by_colbert
Dataset Card for "commonsense_qa_retrieved_by_colbert"
This is the validation set of CSQA, with each question retrieved from ColBERT v2 trained on MS MARCO Passage Ranking (https://downloads.cs.stanford.edu/nlp/data/colbert/colbertv2/colbertv2.0.tar.gz).
We index the question part of the train set using doc_maxlen=30, nbits=2. We search each question of validation set with k=10 and put the results in the retrieved column.
NQ-colbertopenbookqa_retrieved_by_colbert
Dataset Card for "openbookqa_retrieved_by_colbert"
This is the main/test set of OBQA, with each question retrieved from ColBERT v2 trained on MS MARCO Passage Ranking (https://downloads.cs.stanford.edu/nlp/data/colbert/colbertv2/colbertv2.0.tar.gz).
We index the question part of the train set using doc_maxlen=30, nbits=2. We search each question of test set with k=10 and put the results in the retrieved column.
NQ-colbert-20kclini-colbert-pairs-dev-v2NQ-colbert-10kNQ-colbert-top-10NQ-colbert-10k-caseEVAL.ColBERT.IR_evaluationNQ-colbert-10k-case-entityNQ-colbert-10k_preprocessedrank-distillm-rankzephyr_colbert_10000_100-grouped