datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
mmarco-hard-negatives-reranker-filtered
mMARCO Reranker-Filtered Hard Negatives (Multilingual)
Overview
This dataset is built from mMARCO (multilingual MS MARCO) triplets for each language subset. For each (query, positive), hard negatives are bundled and then filtered using cross-encoder re-scoring. The goal is to remove negatives that are too strong or incorrect for training. The same procedure is applied to all language subsets.
The dataset is published as mmarco-hard-negatives-reranker-filtered with… See the full description on the dataset page: https://huggingface.co/datasets/hotchpotch/mmarco-hard-negatives-reranker-filtered.bekko-embedding-v1-hard-negatives
hotchpotch/bekko-embedding-hard-negatives
hotchpotch/bekko-embedding-hard-negatives is a private collection of hard-negative training subsets for information retrieval and embedding-model training. Each row contains one query, one positive text, and 15 mined hard negatives in a flat schema:
query, pos, neg_1, neg_2, ..., neg_15
The dataset is intended for training dense retrievers, sparse/dense hybrid retrievers, embedding models, and rerankers that benefit from explicit… See the full description on the dataset page: https://huggingface.co/datasets/hotchpotch/bekko-embedding-v1-hard-negatives.finqa-hard-negatives
FinQA hard negatives
Near-duplicate hard negatives for training financial-document retrievers, built
from the train split of FinQA
(Chen et al., arXiv:2109.00122).
Each row is one question, its gold evidence unit(s), and the other evidence
units from the same filing page as hard negatives — same company, same
fiscal year, same boilerplate. The distinction a model must learn is which
sentence or table row on the page answers the question, not which company or
topic is relevant.… See the full description on the dataset page: https://huggingface.co/datasets/bowang0911/finqa-hard-negatives.msmarco-hard-negatives
MS MARCO Passages Hard Negatives
[!NOTE]
This repository contains raw datasets, all of which have also been formatted for easy training in the MS MARCO Mined Triplets collection. We recommend looking there first.
MS MARCO is a large scale information retrieval corpus that was created based on real user search queries using Bing search engine.
This dataset repository contains files that are helpful to train bi-encoder models e.g. using sentence-transformers.
Training Code… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/msmarco-hard-negatives.codeedit_hard_negative_datasets_kdmmarco-hard-negatives-reranker-score
hotchpotch/mmarco-hard-negatives-reranker-score
This repository contains data from mMARCO scored using the reranker BAAI/bge-reranker-v2-m3.
Languages Covered
target_languages = [
"english",
"chinese",
"french",
"german",
"indonesian",
"italian",
"portuguese",
"russian",
"spanish",
"arabic",
"dutch",
"hindi",
"japanese",
"vietnamese"
]
Hard Negative Data
The hard negative data is derived from… See the full description on the dataset page: https://huggingface.co/datasets/hotchpotch/mmarco-hard-negatives-reranker-score.llm2vec-gen-echo-rewritten-w-hard-negative
LLM2Vec-Gen
The dataset consists of generations based on the Echo data (Springer et al). The instruction+queries are rewritten in a natural tone using Gemini. The generations are intended to be used for training LLM2Vec-Gen models, serving as the target output for queries.
The negative_question in this dataset are also generated by Gemini. This dataset consists of various splits. Each split corresponds to responses generated by a specific LLM, e.g., Qwen3-4B. The "original" split… See the full description on the dataset page: https://huggingface.co/datasets/McGill-NLP/llm2vec-gen-echo-rewritten-w-hard-negative.hard-negativesjapanese-reranker-v2-hard-negatives-scores
hotchpotch/japanese-reranker-v2-hard-negatives-scores
This dataset was used to train the Japanese Reranker v2 family. It was not generated by Japanese Reranker v2 models.
Unified hard-negative score rows generated from the training sources used for Japanese Reranker v2-family experiments.
This dataset keeps teacher scores as floating-point soft labels. It follows the HPPRC Emb Score style: score/id rows and referenced document text collections are stored separately. Each score… See the full description on the dataset page: https://huggingface.co/datasets/hotchpotch/japanese-reranker-v2-hard-negatives-scores.japanese-splade-v1-hard-negatives日本語 SPLADE v2 の学習に用いたデータセットです。
SPLADE モデルである hotchpotch/japanese-splade-base-v1-mmarco-only, japanese-splade-base-v1_5 を用いてハードネガティブマイニングを行なっています。また BAAI/bge-reranker-v2-m3 を用いたリランカースコアを付与しています。
mqa, mmarco はhpprc/emb のデータを用いています。
mqa の query 作成には MinHash を用い約40万件になるようフィルタしました。
msmarco-ja は hpprc/msmarco-jaのデータを用いています。
ライセンスは、各データセットのライセンスを継承します。
mmarco-japanese-hard-negatives[Under Construction]
This is a repository containing all the queries from the Japanese part of the MMarco dataset, the multilingual version of the MSMarco dataset.
For each query, there are matching hard negatives:
25 of them retrieved by the multilingual e5 base model.
Up to 10 of them retrieved by the basic implementation of BM25 from Japanese in the Anserini library.
hard-negative-tripletsgooaq-hard-negatives
Dataset Card for GooAQ with mined hard negatives
This dataset is a collection of question-answer-negative triplets and question-answer-negative_1...-negative_5 tuples from the GooAQ dataset. See GooAQ for additional information.
This dataset can be used directly with Sentence Transformers to train embedding models.
The negative samples have been automatically mined with all-MiniLM-L6-v2 and:
range_min: 10, i.e. we skip the 10 most similar samples
range_max: 100, i.e. we only look… See the full description on the dataset page: https://huggingface.co/datasets/tomaarsen/gooaq-hard-negatives.owl_code_search_hard_negative_datasets-Pre_kd
Owl Code Search Hard Negative Datasets
Knowledge Distillation (KD) ベースのハードネガティブ付きコード検索データセットです。コード検索モデルShuu12121/CodeSearch-ModernBERT-Crow-v3-large-len1024-Plusを教師モデルとして、各コメントと説明コメントのペアのデータセットから各クエリに対する関数の類似度スコアを計算し、ハードネガティブ（正解に類似しているが不正解の文書）を付与しています。
概要
目的: コード検索モデルの Contrastive Learning / Knowledge Distillation ファインチューニング
言語: Go, Java, JavaScript, PHP, Python, Ruby, Rust, TypeScript（8言語）
総サンプル数: 4,787,740
データサイズ: 8.73 GB（展開後） / 3.37 GB（ダウンロード時）
フォーマット:… See the full description on the dataset page: https://huggingface.co/datasets/Shuu12121/owl_code_search_hard_negative_datasets-Pre_kd.multilingual-codesearch-hard-negativesV2
multilingual-codesearch-hard-negativesV2
このデータセットは、多言語コード検索タスク向けに設計された、ハードネガティブペアを提供します。複数のプログラミング言語から収集した、フィルタ済みのコード／ドックストリング対を基に構成されています。
データセット概要
各サンプルには以下の情報が含まれます。
query_docstring: 関数やメソッドの自然言語による説明
positive_code: 対応する正しいコード実装
hard_negative_code: 類似しているが意味が異なるコード（FAISSによる近傍検索で選定）
similarity_score: クエリとハードネガティブ間のFAISS類似度スコア（内積／コサイン類似度）
付加情報: リポジトリ、ファイルパス、ライセンス情報、言語
対応言語
Python
Java
JavaScript
PHP
Ruby
Go
Rust
データセット作成方法… See the full description on the dataset page: https://huggingface.co/datasets/Shuu12121/multilingual-codesearch-hard-negativesV2.miracl-hard-negatives
MIRACL Hard Negatives (Parquet Format)
This is a Parquet-converted version of mteb/miracl-hard-negatives, compatible with the latest HuggingFace datasets library (4.0+).
Why This Dataset?
The original mteb/miracl-hard-negatives uses a Python script-based loader, which is no longer supported in datasets >= 4.0.0. This dataset provides the same data in standard Parquet format.
Dataset Description
MIRACL (Multilingual Information Retrieval Across a Continuum of… See the full description on the dataset page: https://huggingface.co/datasets/datalama/miracl-hard-negatives.msmarco-ja-hard-negatives
msmarco-ja-hard-negatives
hpprc/msmarco-ja で公開されているMS MARCOの日本語翻訳データに、以下の処理を加えたハードネガティブマイニングをしたものです。また、後段タスクとして、情報検索モデルのSPLADEを学習させたモデルで mMARCO との比較を行いました。
データの処理
翻訳データの正規化、フィルタと選択
正規化
ユニコード正規化 (NFKC)
dataset(query) のフィルタ・選択
dataset-sim のコサイン類似度が0.98以上のものはほぼ同一のケースが多く、距離が近すぎるので取り除く
元の英文を丸々含むデータを取り除く
残ったデータで、一番類似度が近いものを選択
collection の選択
dataset(query) の pod_ids (正例)が対象の場合、query, doc ペアで BAAI/bge-reranker-v2-m3 のスコアを用い、最もスコアが高いもの選択
正例に含まれないデータの場合、同一の id… See the full description on the dataset page: https://huggingface.co/datasets/hotchpotch/msmarco-ja-hard-negatives.guardrail-hard-negatives
Guardrail Hard Negatives (EN/TR)
A false-positive stress test for guardrails. A curated, paired benign/attack dataset for evaluating and training prompt-injection detectors and LLM guardrails. A bilingual false-positive challenge set: benign prompts that look like attacks (security researchers asking about injection, authorized admin actions, quoted payloads, legitimate roleplay) paired against real attacks, so you can measure the false-positive rate your users will actually… See the full description on the dataset page: https://huggingface.co/datasets/3nesdeniz/guardrail-hard-negatives.sec-xbrl-hard-negatives
SEC XBRL hard negatives
~461k query / positive / hard-negative triplets built from the SEC's
Financial Statement Data Sets (DERA), covering 2023Q1–2026Q1, in two
document granularities.
from datasets import load_dataset
load_dataset("bowang0911/sec-xbrl-hard-negatives", "section") # default
load_dataset("bowang0911/sec-xbrl-hard-negatives", "page")
config
rows
negatives
doc chars (p50)
what a document is
section
461,043
2,259,724 (4.9/q)
531
a 9-line window of a… See the full description on the dataset page: https://huggingface.co/datasets/bowang0911/sec-xbrl-hard-negatives.owl_code_search_hard_negative_datasets_V2_kdgooaq_mt_german_5_hard_negatives
GooAQ (Google Answers to Google Questions) question-answer pairs in German with 5 mined hard negatives.
About
This dataset is a collection of ~2M question-answer-negative triplets and question-answer-negative_1...-negative_5 tuples from the machine translated version of MarcGrumpyOlejak/gooaq_mt_german. The full original Gooaq dataset in english only: (link to original dataset). This dataset can be used directly with Sentence Transformers to train embedding models.… See the full description on the dataset page: https://huggingface.co/datasets/MarcGrumpyOlejak/gooaq_mt_german_5_hard_negatives.data_with_hard_negatives_renamedcoir_hard_negative_datasets_v2_kdmsmarco-hard-negatives-scored-stellanatural-questions-hard-negatives
Dataset Card for Natural Questions with mined hard negatives
This dataset is a collection of question-answer-negative triplets and question-answer-negative_1...-negative_5 tuples from the Natural Questions dataset. See Natural Questions for additional information.
This dataset can be used directly with Sentence Transformers to train embedding models.
The negative samples have been automatically mined with all-MiniLM-L6-v2 and:
range_min: 10, i.e. we skip the 10 most similar samples… See the full description on the dataset page: https://huggingface.co/datasets/tomaarsen/natural-questions-hard-negatives.OmniCVR-mini-hard-negativesecho-data-rewritten-queries-hard-negative-qwen3-4bstm_gpt5_hard_negatives
STM GPT-5 Hard Negatives
GPT-5 synthetic hard negatives aligned with the BMRetriever fine-tuning mixture.
Rows align positionally with BMRetriever/biomed_retrieval_dataset (train, 1,429,010 rows).
Column
Description
bmretriever_index
0-based row index in BMRetriever
hard_negative
Synthetic hard-negative passage
from datasets import load_dataset
hn = load_dataset("ikim-uk-essen/stm_gpt5_hard_negatives", split="train")
bm =… See the full description on the dataset page: https://huggingface.co/datasets/ikim-uk-essen/stm_gpt5_hard_negatives.jxm-nomic-hard-negativescodesearchnet-hard-negatives
CodeSearchNet Hard Negatives (Filtered) by Lumees AI
Dataset Summary
This dataset is a processed version of the CodeSearchNet dataset, enhanced with Hard Negative Mining to facilitate the training of state-of-the-art code retrieval models.
It was created by Lumees AI to improve the ability of embedding models to distinguish between syntactically similar but functionally different code snippets.
Developer: Lumees AI
Authors: Hasan Kurşun, Kerem Berkay Yanık… See the full description on the dataset page: https://huggingface.co/datasets/hasankursun/codesearchnet-hard-negatives.