orgrctera/msmarco_passage_ranking

MS MARCO Passage Ranking

Dataset description

MS MARCO (MicroSoft MAchine Reading COmprehension) is a large-scale collection built for machine reading comprehension and information retrieval research. The original release introduced more than one million real user questions sampled from Bing search logs, paired with passages drawn from web documents, and human-authored answers where applicable.

The passage ranking track uses a fixed corpus of short text passages and asks systems to identify which passages are likely to contain an answer to a natural-language query. It is one of the most widely used benchmarks for training and evaluating dense retrieval, late interaction, and reranking models in open-domain QA and neural IR.

This Hugging Face dataset (orgrctera/msmarco_passage_ranking) repackages a dev subset of the MS MARCO passage-ranking task into a simple tabular format for CTERA AI RAG evaluation (benchmark_type: base_rag). Each row is one query; labels reference passage IDs (pid) from the official MS MARCO passage collection, not the passage text inline.

For the authoritative corpus files, qrels, and leaderboard definitions, see the MS MARCO ranking datasets page and the MSMARCO-Passage-Ranking GitHub repository.

Task: passage ranking / retrieval

Passage ranking is an information retrieval task:

• —Input: a query string (often an information need phrased as a question or short phrase).
• —Output: a ranking of passages from a large collection by estimated relevance—specifically, the likelihood that a passage contains information sufficient to answer the query.
• —Typical setup: models score query–passage pairs or build indexable embeddings over the passage collection; full retrieval ranks the entire corpus, while reranking reorders a smaller candidate list (e.g., top-1000 BM25 hits).

Standard leaderboard metrics for MS MARCO passage ranking include MRR@10 (mean reciprocal rank in the top 10) for ranked lists. Training data in the original release includes human relevance judgments (qrels) and, in many pipelines, triples (query, positive passage, negative passage) for contrastive learning.

Important: This Hub dataset stores expected passage IDs as the supervision signal. Full RAG or retrieval experiments still require joining pid to passage text via the official collection.tsv (or another mirror of the MS MARCO passage corpus).

Data format (this repository)

Parquet files with string columns:

Splits and row counts follow the files published under data/ in this dataset repository (e.g., dev split on the Hub).

Examples

Example 1

• —input: cost of endless pools/swim spa
• —expected_output: ["7187234"]
• —metadata (illustrative): {"query_id": "1048578", "split": "dev", "benchmark_name": "msmarco_passage_ranking", "benchmark_type": "base_rag", "sub_benchmark": "passage_ranking"}

Example 2

• —input: what is pcnt
• —expected_output: ["7187227"]
• —metadata (illustrative): {"query_id": "1048579", "split": "dev", "benchmark_name": "msmarco_passage_ranking", "benchmark_type": "base_rag", "sub_benchmark": "passage_ranking"}

References and further reading

MS MARCO (original dataset)

MS MARCO: A Human Generated MAchine Reading COmprehension Dataset — Bajaj et al., 2016. arXiv:1611.09268

Abstract (abridged): We introduce a large scale MAchine Reading COmprehension dataset, which we name MS MARCO. The dataset comprises of 1,010,916 anonymized questions—sampled from Bing's search query logs—each with a human generated answer and 182,669 completely human rewritten generated answers. In addition, the dataset contains 8,841,823 passages—extracted from 3,563,535 web documents retrieved by Bing—that provide the information necessary for curating the natural language answers. … Using this dataset, we propose three different tasks with varying levels of difficulty: … (iii) rank a set of retrieved passages given a question.

BibTeX (from the MS MARCO citation page):

@article{bajaj2016ms,
  title={Ms marco: A human generated machine reading comprehension dataset},
  author={Bajaj, Payal and Campos, Daniel and Craswell, Nick and Deng, Li and Gao, Jianfeng and Liu, Xiaodong and Majumder, Rangan and McNamara, Andrew and Mitra, Bhaskar and Nguyen, Tri and others},
  journal={arXiv preprint arXiv:1611.09268},
  year={2016}
}

Passage ranking / neural baselines

An Updated Duet Model for Passage Re-ranking — Mitra & Craswell, 2019. arXiv:1903.07666

Abstract: We propose several small modifications to Duet—a deep neural ranking model—and evaluate the updated model on the MS MARCO passage ranking task. We report significant improvements from the proposed changes based on an ablation study.

Official resources

• —MS MARCO — Datasets for document and passage ranking leaderboards
• —TREC Deep Learning Track (blind evaluation using MS MARCO-style ranking tasks)
• —MSMARCO-Passage-Ranking (GitHub)

License and terms

The underlying MS MARCO data is subject to the terms and conditions stated by Microsoft (non-commercial research use; see the official site for details). When publishing work that uses MS MARCO or derivatives, cite the MS MARCO paper above and comply with the original license and usage restrictions.

Citation

If you use this Hugging Face dataset, cite MS MARCO (Bajaj et al., 2016) and acknowledge this derivative packaging as appropriate for your publication.