msm
Datasets
All datasets matching “msm”ms_marco
Dataset Card for "ms_marco"
Dataset Summary
Starting with a paper released at NIPS 2016, MS MARCO is a collection of datasets focused on deep learning in search.
The first dataset was a question answering dataset featuring 100,000 real Bing questions and a human generated answer.
Since then we released a 1,000,000 question dataset, a natural langauge generation dataset, a passage ranking dataset,
keyphrase extraction dataset, crawling dataset, and a conversational search.… See the full description on the dataset page: https://huggingface.co/datasets/microsoft/ms_marco.msmarco-beir-e5prebuilt-indexes-msmarco-v1
Prebuilt Indexes for MS MARCO v1
Available indexes:
Lucene Standard Inverted
msmarco-v1-doc
[readme]
Lucene index of the MS MARCO V1 document corpus.
msmarco-v1-doc-slim
[readme]
Lucene index of the MS MARCO V1 document corpus ('slim' version).
msmarco-v1-doc-full
[readme]
Lucene index of the MS MARCO V1 document corpus ('full' version).
msmarco-v1-doc.d2q-t5
[readme]
Lucene index of the MS MARCO V1 document corpus with doc2query-T5 expansions.… See the full description on the dataset page: https://huggingface.co/datasets/castorini/prebuilt-indexes-msmarco-v1.msmarco-v2.1-embed-english-v3
TREC-RAG 2024 Corpus (MSMARCO 2.1) - Encoded with Cohere Embed English v3
This dataset contains the embeddings for the TREC-RAG Corpus 2024 embedded with the Cohere Embed V3 English model.
It contains embeddings for 113,520,750 passages, embeddings for 1677 queries from TREC-Deep Learning 2021-2023, as well as top-1000 hits for all queries using a brute-force (flat) index.
Search over the Index
We have a pre-build index that only requires 300 MB available at… See the full description on the dataset page: https://huggingface.co/datasets/CohereLabs/msmarco-v2.1-embed-english-v3.msmarco-distilbert-margin-mse-mean-dot-v1
MS MARCO with hard negatives from distilbert-margin-mse-mean-dot-v1
MS MARCO is a large scale information retrieval corpus that was created based on real user search queries using the Bing search engine.
For each query and gold positive passage, the 50 most similar paragraphs were mined using 13 different models. The resulting data can be used to train Sentence Transformer models.
Related Datasets
These are the datasets generated using the 13 different models:… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/msmarco-distilbert-margin-mse-mean-dot-v1.msmarco-v2.1-snowflake-arctic-embed-l
Snowflake Arctic Embed L Embeddings for MSMARCO V2.1 for TREC-RAG
This dataset contains the embeddings for the MSMARCO-V2.1 dataset which is used as the corpora for TREC RAG
All embeddings are created using Snowflake's Arctic Embed L and are intended to serve as a simple baseline for dense retrieval-based methods.
Retrieval Performance
Retrieval performance for the TREC DL21-23, MSMARCOV2-Dev and Raggy Queries can be found below with BM25 as a baseline. For both… See the full description on the dataset page: https://huggingface.co/datasets/Snowflake/msmarco-v2.1-snowflake-arctic-embed-l.