datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
msmarco-passagemsmarco
MSMARCO
An MTEB dataset
Massive Text Embedding Benchmark
MS MARCO is a collection of datasets focused on deep learning in search
Task category
t2t
Domains
Encyclopaedic, Academic, Blog, News, Medical, Government, Reviews, Non-fiction, Social, Web
Reference
https://microsoft.github.io/msmarco/
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_tasks(["MSMARCO"])
evaluator… See the full description on the dataset page: https://huggingface.co/datasets/mteb/msmarco.msmarco-passage-corpusms-marco-anchor-text
Webis MS MARCO Anchor Text 2022
The Webis MS MARCO Anchor Text 2022 dataset enriches Version 1 and 2 of the document collection of MS MARCO with anchor text extracted from six Common Crawl snapshots. The six Common Crawl snapshots cover the years 2016 to 2021 (between 1.7-3.4 billion documents each). We sampled 1,000 anchor texts for documents with more than 1,000 anchor texts at random and all anchor texts for documents with less than 1,000 anchor texts (this sampling yields that… See the full description on the dataset page: https://huggingface.co/datasets/webis/ms-marco-anchor-text.msmarco_passage_ranking_corpusThis is the preprocessed data from msmarco passage(v1) ranking corpus.
MS MARCO: A human generated MAchine Reading COmprehension dataset SPayal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen,.
ms-marco-dummy
MS MARCO dummy+test dataset
Used for testing nixietune: a dummy dataset of random 1000 queries from MS MARCO. The format is the following:
{
"query": ")what was the immediate impact of the success of the manhattan project?",
"positive": [
"The presence of communication amid scientific minds was equally important to the success of the Manhattan Project as scientific intellect was. The only cloud hanging over the impressive achievement of the atomic researchers and engineers… See the full description on the dataset page: https://huggingface.co/datasets/nixiesearch/ms-marco-dummy.msmarco-v2
MSMARCOv2
An MTEB dataset
Massive Text Embedding Benchmark
MS MARCO is a collection of datasets focused on deep learning in search
Task category
t2t
Domains
Encyclopaedic, Academic, Blog, News, Medical, Government, Reviews, Non-fiction, Social, Web
Reference
https://microsoft.github.io/msmarco/TREC-Deep-Learning.html
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/msmarco-v2.msmarco
HSEB MSMARCO benchmarking dataset
This collection is based on MSMARCO dataset:
Embedding models:
384 dims: sentence-transformers/all-MiniLM-L6-v2
768 dims: intfloat/e5-base-v2
2560 dims: Qwen3-Embedding-4B
Splits:
1K: 1K documents, 10K queries
100K: 100K documents, 10K queries
1M: 1M documents, 10K queries
Filter selectivity:
10% for high selectivity, 90% for low selectivity, 100% for no filters at all
each document has a tag based on sampled selectivity, so 10% of docs have a… See the full description on the dataset page: https://huggingface.co/datasets/hseb-benchmark/msmarco.hypencoder-msmarco-training-datasetThe MSMARCO training data used to train the models from Hypencoder: Hypernetworks for Information Retrieval
.
Dataset Overview
This dataset is based on the MSMARCO Passage dataset and includes all the queries which have a positive passage in the original dataset (there are additional queries with no positive passages which we do not use). Each query has the known positive passage as well as 200 additional passages. These additional passages may be unlabeled positives or negatives.… See the full description on the dataset page: https://huggingface.co/datasets/jfkback/hypencoder-msmarco-training-dataset.msmarco-2.1-segmentedmsmarco-item-id-hardneg-100shot-v4_128kmsmarco_passage_ranking_official_trainThis is the preprocessed training data from msmarco passage(v1) ranking corpus.
MS MARCO: A human generated MAchine Reading COmprehension dataset SPayal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen,.
rank1-R1-MSMARCO
rank1-R1-MSMARCO: Reasoning Outputs from MS MARCO Dataset
📄 Paper | 🚀 GitHub Repository
This dataset contains outputs from Deepseek's R1 model on the MS MARCO passage dataset, used to train rank1. It showcases the reasoning chains and relevance judgments generated when determining document relevance for information retrieval queries.
Dataset Description
The rank1-R1-MSMARCO dataset consists of reasoning chains and relevance judgments produced on the MS MARCO passage… See the full description on the dataset page: https://huggingface.co/datasets/jhu-clsp/rank1-R1-MSMARCO.msmarco300k-rawv3_msmarco_parallelai_e5qwen7b_6intent_claim_degrademsmarco-atomic-id-3shot-v4_128k_few_shot
msmarco-atomic-id-3shot-v4_128k
MSMARCO few-shot evaluation dataset for in-context learning generative retrieval,
atomic-id variant.
Identical construction to
Lala8383/msmarco-item-id-3shot-v4_128k_few_shot,
except every document's Identifier (and the answer target) is an arbitrary
unique integer (Tay et al. DSI "Atomic Docid") instead of the natural-language
document title. The id carries no semantics, so a retriever can only answer by
matching the query to a document in… See the full description on the dataset page: https://huggingface.co/datasets/Lala8383/msmarco-atomic-id-3shot-v4_128k_few_shot.fineweb_v1_msmarco_parallelai_e5qwen7b_6intent_claim_degradems-marco-hard-negatives
MS MARCO hard negatives dataset
A dataset in a nixietune compatible format:
{
"query": ")what was the immediate impact of the success of the manhattan project?",
"pos": [
"The presence of communication amid scientific minds was equally important to the success of the Manhattan Project as scientific intellect was. The only cloud hanging over the impressive achievement of the atomic researchers and engineers is what their success truly meant; hundreds of thousands of innocent… See the full description on the dataset page: https://huggingface.co/datasets/nixiesearch/ms-marco-hard-negatives.msmarco-chunkeval-original_2000_endRankGPT-msmarco-100kmsmarco-item-id-hardneg-filter-20shot-v4MSMarcomsmarco_passage_ranking_queriesThis is the preprocessed queries from msmarco passage(v1) ranking corpus.
MS MARCO: A human generated MAchine Reading COmprehension dataset SPayal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen,.
msmarco-trMSMARCO_FA_test_top_250_only_w_correct-v2msmlike_v3_msmarco_parallelai_e5qwen7b_5intent_claim_degrademsmarco-chunkeval-id_4000_startmsmarco-chunkeval-200kmsmarco-chunkeval-lorem_ipsum_4000_startms-marco-tr-hard-negatives
MS MARCO TR - Hard Negatives Dataset
Dataset Summary
This dataset contains Hard Negatives specifically mined for the Turkish MS MARCO dataset. It is designed for training or fine-tuning sentence embedding models (e.g., SBERT) for Turkish Information Retrieval tasks.
[Image of vector space diagram showing query positive hard negative and random negative]
Unlike standard random negatives, these "hard" negatives are passages that share high semantic similarity (high vector… See the full description on the dataset page: https://huggingface.co/datasets/lumees/ms-marco-tr-hard-negatives.