datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
finqa-hard-negatives
FinQA hard negatives
Near-duplicate hard negatives for training financial-document retrievers, built
from the train split of FinQA
(Chen et al., arXiv:2109.00122).
Each row is one question, its gold evidence unit(s), and the other evidence
units from the same filing page as hard negatives — same company, same
fiscal year, same boilerplate. The distinction a model must learn is which
sentence or table row on the page answers the question, not which company or
topic is relevant.… See the full description on the dataset page: https://huggingface.co/datasets/bowang0911/finqa-hard-negatives.codesearchnet-hard-negatives
CodeSearchNet Hard Negatives (Filtered) by Lumees AI
Dataset Summary
This dataset is a processed version of the CodeSearchNet dataset, enhanced with Hard Negative Mining to facilitate the training of state-of-the-art code retrieval models.
It was created by Lumees AI to improve the ability of embedding models to distinguish between syntactically similar but functionally different code snippets.
Developer: Lumees AI
Authors: Hasan Kurşun, Kerem Berkay Yanık… See the full description on the dataset page: https://huggingface.co/datasets/hasankursun/codesearchnet-hard-negatives.opengloss-v1.3-hard-negative-pairs
See also OpenGloss v2.1 (2026-09-07): a deeper release of 109,633 of these headwords — sense-level ids, four reading levels, sense-tagged examples with spans, a judged relation graph, and retrieval supervision — published as a 16-dataset family. v1.3 remains the broader headword list.
OpenGloss Hard Negative Pairs v1.3
This dataset contains calibration-oriented positive and low-label similarity pairs for embedding training.
It is designed to reduce over-scoring of… See the full description on the dataset page: https://huggingface.co/datasets/mjbommar/opengloss-v1.3-hard-negative-pairs.amazon-esci-hardnegatives
Amazon ESCI hard-negatives dataset
A dataset in a nixietune compatible format:
{
{
"query": "# cellist thats not a hashtag",
"pos": [
"Funny Cellists That's Not A Hashtag Music Sweatshirt",
"Marvel Deadpool Crunch Cereal Comics Funny Adult Men’s Graphic T-Shirt (Black, Medium)",
"Womens Funny Cellists That's Not A Hashtag Music V-Neck T-Shirt",
"Cellist Gift Orchestra Conductor Thats A Sharp Not A Hashtag T-Shirt",
],
"neg": [
"Feelin Good Tees My Opinion… See the full description on the dataset page: https://huggingface.co/datasets/nixiesearch/amazon-esci-hardnegatives.ms-marco-hard-negatives
MS MARCO hard negatives dataset
A dataset in a nixietune compatible format:
{
"query": ")what was the immediate impact of the success of the manhattan project?",
"pos": [
"The presence of communication amid scientific minds was equally important to the success of the Manhattan Project as scientific intellect was. The only cloud hanging over the impressive achievement of the atomic researchers and engineers is what their success truly meant; hundreds of thousands of innocent… See the full description on the dataset page: https://huggingface.co/datasets/nixiesearch/ms-marco-hard-negatives.hard-negatives-traversal
If our work was helpful conside citing us ☺️
@misc{sinha2025bicaeffectivebiomedicaldense,
title={BiCA: Effective Biomedical Dense Retrieval with Citation-Aware Hard Negatives},
author={Aarush Sinha and Pavan Kumar S and Roshan Balaji and Nirav Pravinbhai Bhatt},
year={2025},
eprint={2511.08029},
archivePrefix={arXiv},
primaryClass={cs.IR},
url={https://arxiv.org/abs/2511.08029},
}
tinyperson-yolov8n-p2p3p4-hard-negative-mosaic-runsms-marco-tr-hard-negatives
MS MARCO TR - Hard Negatives Dataset
Dataset Summary
This dataset contains Hard Negatives specifically mined for the Turkish MS MARCO dataset. It is designed for training or fine-tuning sentence embedding models (e.g., SBERT) for Turkish Information Retrieval tasks.
[Image of vector space diagram showing query positive hard negative and random negative]
Unlike standard random negatives, these "hard" negatives are passages that share high semantic similarity (high vector… See the full description on the dataset page: https://huggingface.co/datasets/lumees/ms-marco-tr-hard-negatives.levir-yolov8n-p2p3p4-hard-negative-mosaic-runsstatute-disease-hard-negatives
Attribute-differentiated hard negatives: statutes and disease
Three retrieval training sets built on the same principle: the negatives are
near-duplicates of the gold that differ on exactly one attribute, so the
only way to rank the gold first is to resolve that attribute. Topical
similarity is not enough to separate them, which is the point -- a retriever
trained on topically-separable data never learns fine-grained instance
discrimination.
config
rows
negs/row
query
BM25… See the full description on the dataset page: https://huggingface.co/datasets/bowang0911/statute-disease-hard-negatives.fever-hard-negatives
Dataset Card
Dataset Details
This dataset contains a set of candidate documents for second-stage re-ranking on fever
(test split in BEIR). Those candidate documents are composed of hard negatives mined from
gtr-t5-xl as Stage 1 ranker
and ground-truth documents that are known to be relevant to the query. This is a release from our paper
Policy-Gradient Training of Language Models for Ranking, so
please cite it if using this dataset.
Direct Use
You can… See the full description on the dataset page: https://huggingface.co/datasets/NeuralPGRank/fever-hard-negatives.beir-eval-hard-negatives
BEIR/MTEB hard negatives dataset
A dataset for quick evaluation of embedding models during their training.
The problem: running a full MTEB evaluation on a single GPU may take 10-20 hours. Most of this time is spent on embedding all 30M docs in all 10+ corpora. This dataset solves this problem by unwrapping a "retrieval" style benchmark into the "reranking" style:
We compute embeddings for all documents in the corpora with the intfloat/e5-base-v2 model.
For each corpus in BEIR/MTEB… See the full description on the dataset page: https://huggingface.co/datasets/nixiesearch/beir-eval-hard-negatives.msmarco_hard_negatives_phi4-14bThe data was used in the paper Don't Retrieve, Generate: Prompting LLMs for Synthetic Training Data in Dense Retrieval.
Hard-Negatives generated by Phi4-14B using 10,000 passages from the MS-Marco dataset
If this dataset was useful consider citing us :)
@misc{sinha2025dontretrievegenerateprompting,
title={Don't Retrieve, Generate: Prompting LLMs for Synthetic Training Data in Dense Retrieval},
author={Aarush Sinha},
year={2025},
eprint={2504.21015}… See the full description on the dataset page: https://huggingface.co/datasets/chungimungi/msmarco_hard_negatives_phi4-14b.msmarco_hard_negatives_qwen3-30bThe data was used in the paper Don't Retrieve, Generate: Prompting LLMs for Synthetic Training Data in Dense Retrieval.
Hard-Negatives generated by Qwen3-30B model using 10,000 passages from the MS-Marco dataset
If this dataset was useful consider citing us :)
@misc{sinha2025dontretrievegenerateprompting,
title={Don't Retrieve, Generate: Prompting LLMs for Synthetic Training Data in Dense Retrieval},
author={Aarush Sinha},
year={2025},
eprint={2504.21015}… See the full description on the dataset page: https://huggingface.co/datasets/chungimungi/msmarco_hard_negatives_qwen3-30b.ms-marco-qwen3-30b-hard-negativesopengloss-v1.3-hard-negative-pairs
OpenGloss Hard Negative Pairs v1.3
This dataset contains calibration-oriented positive and low-label similarity pairs for embedding training.
It is designed to reduce over-scoring of related-but-wrong matches and improve score separation in weak domains.
Dataset Summary
Total records: 1,131,241
Unique lexemes: 205,967
Relation Distribution
Relation Type
Count
same_domain_wrong_entity
566,913
style_variant
205,963
true_match
205,956… See the full description on the dataset page: https://huggingface.co/datasets/Cloudadorablebearcloudbear/opengloss-v1.3-hard-negative-pairs.quora-hard-negatives
Dataset Card
Dataset Details
This dataset contains a set of candidate documents for second-stage re-ranking on quora
(test split in BEIR). Those candidate documents are composed of hard negatives mined from
gtr-t5-xl as Stage 1 ranker
and ground-truth documents that are known to be relevant to the query. This is a release from our paper
Policy-Gradient Training of Language Models for Ranking, so
please cite it if using this dataset.
Direct Use
You can… See the full description on the dataset page: https://huggingface.co/datasets/NeuralPGRank/quora-hard-negatives.hard-negatives-ru
hard-negatives-ru
Russian near-miss retrieval pairs, machine-verified. 8470 question–chunk pairs for
training a cross-encoder reranker, 2967 conversational paraphrases, and a separate
498-item twins evaluation set on which a zero-false-accept threshold was measured
and found to cost 94 % of recall.
The dataset exists because of one measurement, and that measurement is the reason to
use it:
Over 8470 pairs where a program has proved whether the chunk answers the question,
the… See the full description on the dataset page: https://huggingface.co/datasets/digitable-lol/hard-negatives-ru.arxiv-hard-negative-LLMThis dataset utilizes an LLM to generate both a query and a hard negative example from a given passage.
miracl-hard-negatives-mini
MIRACL Hard Negatives Mini
Minimized version of mteb/MIRACLRetrievalHardNegatives for rapid testing.
100 Queries per language (max)
5000 Documents per language (max)
⚠️ Sampling Note
The documents in this dataset are a stratified random subset of the original mteb/MIRACLRetrievalHardNegatives corpus.
We ensure all ground-truth documents for the selected queries are included.
The remaining documents are randomly sampled from the original corpus to maintain the "hard… See the full description on the dataset page: https://huggingface.co/datasets/datalama/miracl-hard-negatives-mini.dbpedia-entity-hard-negatives
Dataset Card
Dataset Details
This dataset contains a set of candidate documents for second-stage re-ranking on dbpedia
(test split in BEIR). Those candidate documents are composed of hard negatives mined from
gtr-t5-xl as Stage 1 ranker
and ground-truth documents that are known to be relevant to the query. This is a release from our paper
Policy-Gradient Training of Language Models for Ranking, so
please cite it if using this dataset.
Direct Use
You can… See the full description on the dataset page: https://huggingface.co/datasets/NeuralPGRank/dbpedia-entity-hard-negatives.scidocs-hard-negatives
Dataset Card
Dataset Details
This dataset contains a set of candidate documents for second-stage re-ranking on scidocs
(test split in BEIR). Those candidate documents are composed of hard negatives mined from
gtr-t5-xl as Stage 1 ranker
and ground-truth documents that are known to be relevant to the query. This is a release from our paper
Policy-Gradient Training of Language Models for Ranking, so
please cite it if using this dataset.
Direct Use
You can… See the full description on the dataset page: https://huggingface.co/datasets/NeuralPGRank/scidocs-hard-negatives.arxiv-hard-negatives-bm25Hard Negatives mined using BM25 from a corpus of 7249 papers from arxiv with top-k @ k=50
arxiv-hard-negatives-cross-encoderThis dataset contains hard negative examples generated using cross-encoders for training dense retrieval models.
@misc{reimers2019sentencebertsentenceembeddingsusing,
title={Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks},
author={Nils Reimers and Iryna Gurevych},
year={2019},
eprint={1908.10084},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/1908.10084},
}
msmarco_hard_negatives_qwen3-4bThe data was used in the paper Don't Retrieve, Generate: Prompting LLMs for Synthetic Training Data in Dense Retrieval.
Hard-Negatives generated by Qwen3-4B model using 10,000 passages from the MS-Marco dataset
If this dataset was useful consider citing us :)
@misc{sinha2025dontretrievegenerateprompting,
title={Don't Retrieve, Generate: Prompting LLMs for Synthetic Training Data in Dense Retrieval},
author={Aarush Sinha},
year={2025},
eprint={2504.21015}… See the full description on the dataset page: https://huggingface.co/datasets/chungimungi/msmarco_hard_negatives_qwen3-4b.msmarco_hard_negatives_cross-encoderThe data was used in the paper Don't Retrieve, Generate: Prompting LLMs for Synthetic Training Data in Dense Retrieval.
Hard-Negatives generated by a cross-encoder using 10,000 passages from the MS-Marco dataset
If this dataset was useful consider citing us :)
@misc{sinha2025dontretrievegenerateprompting,
title={Don't Retrieve, Generate: Prompting LLMs for Synthetic Training Data in Dense Retrieval},
author={Aarush Sinha},
year={2025},
eprint={2504.21015}… See the full description on the dataset page: https://huggingface.co/datasets/chungimungi/msmarco_hard_negatives_cross-encoder.nq-hard-negatives
Dataset Card
Dataset Details
This dataset contains a set of candidate documents for second-stage re-ranking on nq
(test split in BEIR). Those candidate documents are composed of hard negatives mined from
gtr-t5-xl as Stage 1 ranker
and ground-truth documents that are known to be relevant to the query. This is a release from our paper
Policy-Gradient Training of Language Models for Ranking, so
please cite it if using this dataset.
Direct Use
You can load… See the full description on the dataset page: https://huggingface.co/datasets/NeuralPGRank/nq-hard-negatives.opengloss-v1.2-hard-negative-pairs
OpenGloss Hard Negative Pairs v1.2
This dataset contains calibration-oriented positive and low-label similarity pairs for embedding training.
It is designed to reduce over-scoring of related-but-wrong matches and improve score separation in weak domains.
Dataset Summary
Total records: 73,244
Unique lexemes: 11,522
Relation Distribution
Relation Type
Count
same_domain_wrong_entity
27,216
near_fact_confusion
22,095
style_variant
11,519… See the full description on the dataset page: https://huggingface.co/datasets/mjbommar/opengloss-v1.2-hard-negative-pairs.ms-marco-bm25-hard-negativesclimate-fever-hard-negatives
Dataset Card
Dataset Details
This dataset contains a set of candidate documents for second-stage re-ranking on climate-fever
(test split in BEIR). Those candidate documents are composed of hard negatives mined from
gtr-t5-xl as Stage 1 ranker
and ground-truth documents that are known to be relevant to the query. This is a release from our paper
Policy-Gradient Training of Language Models for Ranking, so
please cite it if using this dataset.
Direct Use… See the full description on the dataset page: https://huggingface.co/datasets/NeuralPGRank/climate-fever-hard-negatives.