datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
msmarco-v2.1-snowflake-arctic-embed-l
Snowflake Arctic Embed L Embeddings for MSMARCO V2.1 for TREC-RAG
This dataset contains the embeddings for the MSMARCO-V2.1 dataset which is used as the corpora for TREC RAG
All embeddings are created using Snowflake's Arctic Embed L and are intended to serve as a simple baseline for dense retrieval-based methods.
Retrieval Performance
Retrieval performance for the TREC DL21-23, MSMARCOV2-Dev and Raggy Queries can be found below with BM25 as a baseline. For both… See the full description on the dataset page: https://huggingface.co/datasets/Snowflake/msmarco-v2.1-snowflake-arctic-embed-l.msmarco-v2.1-snowflake-arctic-embed-m-v1.5
Snowflake Arctic Embed M V1.5 Embeddings for MSMARCO V2.1 for TREC-RAG
This dataset contains the embeddings for the MSMARCO-V2.1 dataset which is used as the corpora for TREC RAG
All embeddings are created using Snowflake's Arctic Embed M v1.5 and are intended to serve as a simple baseline for dense retrieval-based methods.
It's worth noting that Snowflake's Arctic Embed M v1.5 is optimized for efficient embeddings and thus supports embedding truncation and quantization. More… See the full description on the dataset page: https://huggingface.co/datasets/Snowflake/msmarco-v2.1-snowflake-arctic-embed-m-v1.5.MSMarco-ES-TTS-Big95.1ksamples
MSMarco-ES-TTS-Big95.1ksamples 🎙️📖
Extensión del dataset ejbejaranos/MSMarco-ES-TTS-small4.5ksamples que contiene 95,108 muestras adicionales de pares pregunta-respuesta en español convertidos a audio mediante síntesis de voz (TTS). Los samples son distintos a los de la versión anterior, por lo que pueden usarse como extensión.
🔍 Vista rápida
🎧 Demostración de audio
Audio de instrucción (pregunta):
Tu navegador no soporta audio HTML5.
Audio… See the full description on the dataset page: https://huggingface.co/datasets/ejbejaranos/MSMarco-ES-TTS-Big95.1ksamples.ms-marco-es
Dataset Card for "ms-marco-es"
QA asymmetric Spanish dataset filtered from multilingual version of MS Marco
import datasets
ms_marco_es = datasets.load_dataset('unicamp-dl/mmarco', name='spanish', split='train')
ms_marco_es.push_to_hub("dariolopez/ms-marco-es", token=os.environ['hg_token'])
ms-marco-v2.1-lance
MS MARCO v2.1 QA (Lance Format)
A Lance-formatted version of MS MARCO v2.1 — Microsoft's machine-reading-comprehension benchmark built from anonymized Bing query logs. Each row is one user query, the up-to-10 candidate passages Bing retrieved for it with relevance flags, and the human-written reference answers, with MiniLM query embeddings stored inline and pre-built ANN/FTS indices, available directly from the Hub at hf://datasets/lance-format/ms-marco-v2.1-lance/data.… See the full description on the dataset page: https://huggingface.co/datasets/lance-format/ms-marco-v2.1-lance.msmarco-v2.1-stella_en_1.5B_v5
NovaSearch stella_en_1.5B_v5 Embeddings for MSMARCO V2.1 for TREC-RAG
This dataset contains the embeddings for the MSMARCO-V2.1 dataset which is used as the corpora for TREC RAG
All embeddings are created using Stella EN 1.5B V5 and are intended to serve as a simple baseline for dense retrieval-based methods.
Note, that the embeddings are not normalized so you will need to normalize them before usage.
Retrieval Performance
Retrieval performance for the TREC DL21-23… See the full description on the dataset page: https://huggingface.co/datasets/spacemanidol/msmarco-v2.1-stella_en_1.5B_v5.msmarco-tr
Dataset Card for "msmarco-tr"
More Information needed
msmarco-v2.1-gte-large-en-v1.5
Alibaba GTE-Large-V1.5 Embeddings for MSMARCO V2.1 for TREC-RAG
This dataset contains the embeddings for the MSMARCO-V2.1 dataset which is used as the corpora for TREC RAG
All embeddings are created using GTE Large V1.5 and are intended to serve as a simple baseline for dense retrieval-based methods.
Note, that the embeddings are not normalized so you will need to normalize them before usage.
Retrieval Performance
Retrieval performance for the TREC DL21-23… See the full description on the dataset page: https://huggingface.co/datasets/spacemanidol/msmarco-v2.1-gte-large-en-v1.5.msmarco-v2.1-snowflake-arctic-embed-m-v2.0
Snowflake Arctic Embed M V2.0 Embeddings for MSMARCO V2.1 for TREC-RAG
This dataset contains the embeddings for the MSMARCO-V2.1 dataset which is used as the corpora for TREC RAG
All embeddings are created using Snowflake's Arctic Embed M v2.0 and are intended to serve as a simple baseline for dense retrieval-based methods.
Note, that the embeddings are not normalized so you will need to normalize them before usage.
Retrieval Performance
Retrieval performance for… See the full description on the dataset page: https://huggingface.co/datasets/Snowflake/msmarco-v2.1-snowflake-arctic-embed-m-v2.0.MS-MARCO-0G-120M
MS-MARCO-0G-120M
A long-context retrieval and question-answering benchmark with 1,143,371 documents and a repaired 192-question default evaluation (128 + 64), including 12 questions about 0G. The original 10,000-question construction pool (including 50 0G questions) is also retained. It extends the ms_100M bank from MSA-RAG-BENCHMARKS with 20 million additional text tokens and 655 new questions grounded in the added documents.
120M is the benchmark's nominal size. The actual… See the full description on the dataset page: https://huggingface.co/datasets/0G-AI/MS-MARCO-0G-120M.hypencoder-msmarco-training-datasetThe MSMARCO training data used to train the models from Hypencoder: Hypernetworks for Information Retrieval
.
Dataset Overview
This dataset is based on the MSMARCO Passage dataset and includes all the queries which have a positive passage in the original dataset (there are additional queries with no positive passages which we do not use). Each query has the known positive passage as well as 200 additional passages. These additional passages may be unlabeled positives or negatives.… See the full description on the dataset page: https://huggingface.co/datasets/jfkback/hypencoder-msmarco-training-dataset.urdu-msmarco-dataset
Dataset Card for Dataset Name
This dataset is the translation of the MS-marco dataset, marking it the first large-scale urdu IR dataset.
Dataset Details
The MS MARCO dataset is formed by a collection of 8.8M passages, approximately 530k queries, and at least one relevant passage per query, which were selected by humans.
The development set of MS MARCO comprises more than 100k queries. However, a smaller set of 6,980 queries is used for evaluation in most published works.… See the full description on the dataset page: https://huggingface.co/datasets/Mavkif/urdu-msmarco-dataset.MSMarco-ES-TTS-small4.5ksamples
MSMarco-ES-TTS-small4.5ksamples 🎙️📚
Este dataset contiene 4,500 muestras de consultas y respuestas en español generadas a partir del dataset MS Marco ES y convertidas a audio mediante un sistema de Text-to-Speech (TTS).
Ejemplo de audio
Puedes escuchar un ejemplo de audio generado a partir del dataset aquí:
Instruction audio
Tu navegador no soporta la reproducción de audio.
Output audio
Descripción
El conjunto de datos… See the full description on the dataset page: https://huggingface.co/datasets/ejbejaranos/MSMarco-ES-TTS-small4.5ksamples.MSMARCO-PL
MSMARCO-PL
An MTEB dataset
Massive Text Embedding Benchmark
MS MARCO is a collection of datasets focused on deep learning in search
Task category
t2t
Domains
Web, Written
Reference
https://microsoft.github.io/msmarco/
Source datasets:
mteb/msmarco
mteb/MSMARCO-PL
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import mteb
task = mteb.get_task("MSMARCO-PL")
evaluator = mteb.MTEB([task])… See the full description on the dataset page: https://huggingface.co/datasets/mteb/MSMARCO-PL.ms_marco_care
Dataset Card for MS MARCO for CARE Training
Dataset Description
This dataset is a processed version of MS MARCO adapted for training language models with native retrieval-augmented reasoning capabilities in the CARE framework. It serves as the "hard" dataset (D_hard) in the curriculum learning strategy for reinforcement learning training.
Key Features
808,731 training instances, 101,093 validation instances, and 101,092 test instances… See the full description on the dataset page: https://huggingface.co/datasets/sheryc/ms_marco_care.msmarco-nlgen
Dataset Card for MSMARCO - Natural Language Generation Task
Dataset Summary
The original focus of MSMARCO was to provide a corpus for training and testing systems which given a real domain user query systems would then provide the most likley candidate answer and do so in language which was natural and conversational. All questions have been generated from real anonymized Bing user queries which grounds the dataset in a real world problem and can provide researchers real… See the full description on the dataset page: https://huggingface.co/datasets/din0s/msmarco-nlgen.nano-msmarco-vn
NanoMSMARCO-VN
An MTEB dataset
Massive Text Embedding Benchmark
A translated dataset from MS MARCO is a collection of datasets focused on deep learning in search The process of creating the VN-MTEB (Vietnamese Massive Text Embedding Benchmark) from English samples involves a new automated system: - The system uses large language models (LLMs), specifically Coherence's Aya model, for translation. - Applies advanced embedding models to filter the translations. - Use LLM-as-a-judge to… See the full description on the dataset page: https://huggingface.co/datasets/GreenNode/nano-msmarco-vn.ms-marco-qa-10k
MS MARCO QA Subset (10K)
This is a subset of the MS MARCO v1.1 dataset by Microsoft, sampled for lightweight experimentation.
Source
Original dataset: microsoft/ms_marco (v1.1)
Original paper: MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Original authors: Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, Li Deng (Microsoft)
What was changed
Randomly sampled 10,000 examples from the train split… See the full description on the dataset page: https://huggingface.co/datasets/Lala8383/ms-marco-qa-10k.ms-marco-es-500k
Dataset Card for "ms-marco-es-500k"
QA asymmetric Spanish dataset filtered from multilingual version of MS Marco and sampled on 500k rows.
import datasets
ms_marco_es = datasets.load_dataset('unicamp-dl/mmarco', name='spanish', split='train')
ms_marco_es.select(range(500_000)).push_to_hub("dariolopez/ms-marco-es-500k", token=os.environ['hg_token'])
ms_marco_sr
Dataset Card for Serbian MS MARCO (Subset)
Dataset Summary
This dataset is a Serbian translation of the first 8,000 examples from Microsoft's MS MARCO (Machine Reading Comprehension) dataset. It contains pairs of questions and human-generated answers, automatically translated from English to Serbian. The dataset is designed for evaluating embedding models on Question Answering (QA) and Information Retrieval (IR) tasks in the Serbian language.
The original MS MARCO dataset… See the full description on the dataset page: https://huggingface.co/datasets/smartcat/ms_marco_sr.ragu-webq-contriever-msmarco
RAGU WebQuestions retrieval data
This repository contains WebQuestions train/dev/test splits prepared for the
RAGU codebase. Each record contains question, answers, q_id, and ctxs.
Retrieval provenance
Retriever: facebook/contriever-msmarco
Similarity: exact flat inner product
Knowledge base: DPR 2018 English Wikipedia, 100-word passages
Retrieved contexts retained per question: 20
WebQ source: official DPR resources
Retrieval was performed shard by shard to… See the full description on the dataset page: https://huggingface.co/datasets/danny2507/ragu-webq-contriever-msmarco.MS_MARCO_Gold_Passage_QA
How to Make
Make "Question has only one answer".
If Question has multiple answer, that question is deleted.
Example Usage
from datasets import load_dataset
docs = load_dataset('jun000/MS_MARCO_Gold_Passage_QA')
msmarco-ICL-100k
In-Context Learning Dataset for MSMARCO-100k
train.jsonl
Contains ~150k indexing (doc, docid) & retrieval (query, docid) pairs
test.jsonl
Contains ~20k unseen retrieval pairs
icl_test.jsonl
Contains ~10k unseen indexing & retrieval pairs