msmarco
Datasets
All datasets matching “msmarco”ms_marco
Dataset Card for "ms_marco"
Dataset Summary
Starting with a paper released at NIPS 2016, MS MARCO is a collection of datasets focused on deep learning in search.
The first dataset was a question answering dataset featuring 100,000 real Bing questions and a human generated answer.
Since then we released a 1,000,000 question dataset, a natural langauge generation dataset, a passage ranking dataset,
keyphrase extraction dataset, crawling dataset, and a conversational search.… See the full description on the dataset page: https://huggingface.co/datasets/microsoft/ms_marco.msmarco-beir-e5prebuilt-indexes-msmarco-v1
Prebuilt Indexes for MS MARCO v1
Available indexes:
Lucene Standard Inverted
msmarco-v1-doc
[readme]
Lucene index of the MS MARCO V1 document corpus.
msmarco-v1-doc-slim
[readme]
Lucene index of the MS MARCO V1 document corpus ('slim' version).
msmarco-v1-doc-full
[readme]
Lucene index of the MS MARCO V1 document corpus ('full' version).
msmarco-v1-doc.d2q-t5
[readme]
Lucene index of the MS MARCO V1 document corpus with doc2query-T5 expansions.… See the full description on the dataset page: https://huggingface.co/datasets/castorini/prebuilt-indexes-msmarco-v1.msmarco-distilbert-margin-mse-mean-dot-v1
MS MARCO with hard negatives from distilbert-margin-mse-mean-dot-v1
MS MARCO is a large scale information retrieval corpus that was created based on real user search queries using the Bing search engine.
For each query and gold positive passage, the 50 most similar paragraphs were mined using 13 different models. The resulting data can be used to train Sentence Transformer models.
Related Datasets
These are the datasets generated using the 13 different models:… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/msmarco-distilbert-margin-mse-mean-dot-v1.msmarco-v2.1-embed-english-v3
TREC-RAG 2024 Corpus (MSMARCO 2.1) - Encoded with Cohere Embed English v3
This dataset contains the embeddings for the TREC-RAG Corpus 2024 embedded with the Cohere Embed V3 English model.
It contains embeddings for 113,520,750 passages, embeddings for 1677 queries from TREC-Deep Learning 2021-2023, as well as top-1000 hits for all queries using a brute-force (flat) index.
Search over the Index
We have a pre-build index that only requires 300 MB available at… See the full description on the dataset page: https://huggingface.co/datasets/CohereLabs/msmarco-v2.1-embed-english-v3.MSMARCO-XI
MS MARCO Translations Dataset
Dataset Description
This dataset contains the MS MARCO dataset translated into various Indic languages. The original MS MARCO dataset is a collection of queries, passages, and answers for machine reading comprehension and question answering tasks. Each example includes both the original English content and the translated content, along with translation metadata.
Supported Languages
Language Code
Language Name… See the full description on the dataset page: https://huggingface.co/datasets/ai4bharat/MSMARCO-XI.