CoolFace
Datasetpublic

sentence-transformers/msmarco-corpus

MS MARCO Corpus This dataset allows for a convenient mapping from MS MARCO query/passage ID to the query/passage text. This passage corpus was downloaded from https://msmarco.z22.web.core.windows.net/msmarcoranking/collection.tar.gz, and the queries from https://msmarco.blob.core.windows.net/msmarcoranking/queries.tar.gz (via Wayback Machine). Usage This dataset was designed to allow you to perform the following: from datasets import load_dataset query_dataset =… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/msmarco-corpus.

sourceHugging Faceupdated 2y agoView on Hugging Face
0likes247downloads
Dataset Card

MS MARCO Corpus

This dataset allows for a convenient mapping from MS MARCO query/passage ID to the query/passage text. This passage corpus was downloaded from https://msmarco.z22.web.core.windows.net/msmarcoranking/collection.tar.gz, and the queries from https://msmarco.blob.core.windows.net/msmarcoranking/queries.tar.gz (via Wayback Machine).

Usage

This dataset was designed to allow you to perform the following:

python
from datasets import load_dataset

query_dataset = load_dataset("sentence-transformers/msmarco-corpus", "query", split="train")
qid_to_query = dict(zip(query_dataset["qid"], query_dataset["text"]))
print(qid_to_query[571018])
# => "what are the liberal arts?"

passage_dataset = load_dataset("sentence-transformers/msmarco-corpus", "passage", split="train")
pid_to_passage = dict(zip(passage_dataset["pid"], passage_dataset["text"]))
print(pid_to_passage[7349777])
# => "liberal arts. 1. the academic course of instruction at a college intended to provide general knowledge and comprising the arts, humanities, natural sciences, and social sciences, as opposed to professional or technical subjects."

Related Datasets

This dataset is used for the query and passage texts in the following datasets containing MS MARCO with mined hard negatives.