datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
ruri-dataset-reranker
Ruri-Dataset Reranker
Datasets used for training Ruri-Reranker.
Please refer to https://huggingface.co/datasets/hpprc/emb for individual datasets.
ettin-reranker-v1-data
Ettin Reranker v1 Training Data
This is the training dataset used to produce the cross-encoder/ettin-reranker-{17m,32m,68m,150m,400m,1b}-v1 family of CrossEncoder rerankers. It's a mix of broad-domain text-pair data and retrieval pairs rescored with a strong teacher reranker, with every label produced by an automated scoring system rather than a human annotator.
Structure
Every config has the same three columns:
column
type
description
query
string
The… See the full description on the dataset page: https://huggingface.co/datasets/cross-encoder/ettin-reranker-v1-data.mmarco-hard-negatives-reranker-filtered
mMARCO Reranker-Filtered Hard Negatives (Multilingual)
Overview
This dataset is built from mMARCO (multilingual MS MARCO) triplets for each language subset. For each (query, positive), hard negatives are bundled and then filtered using cross-encoder re-scoring. The goal is to remove negatives that are too strong or incorrect for training. The same procedure is applied to all language subsets.
The dataset is published as mmarco-hard-negatives-reranker-filtered with… See the full description on the dataset page: https://huggingface.co/datasets/hotchpotch/mmarco-hard-negatives-reranker-filtered.reranker-scores
Reranker-Scores
既存の日本語検索・QAデータセットについて、データセット中のクエリに付与された正・負例の関連度を多言語・日本語reranker 5種類を用いてスコア付けしたデータセットです。クエリごとに200件程度の正・負例文書が付与されています (事例ごとに付与個数はバラバラなのでご注意ください)。score.pos.avg および score.neg.avg はそれぞれ正例・負例についての5種のrerankerの平均スコアになっています。
Short Name
Hub ID
bge
BAAI/bge-reranker-v2-m3
gte
Alibaba-NLP/gte-multilingual-reranker-base
ruri
cl-nagoya/ruri-reranker-large
ruriv3-preview
cl-nagoya/ruri-v3-reranker-310m-preview
ja-ce… See the full description on the dataset page: https://huggingface.co/datasets/hpprc/reranker-scores.KaLM-reranker-training-data
Lychee-KaLM-Reranker Training Data
A large-scale, ready-to-use multilingual dataset for fine-tuning reranking models.
This repository contains 3,885,265 training samples collected from 54 datasets, covering English, Chinese, and multilingual retrieval tasks. Each sample includes task instructions, positive passages, at least 16 hard negatives, and teacher scores annotated by Qwen3-Reranker-8B.
When expanded into point-wise query–passage pairs, the dataset provides at least 66… See the full description on the dataset page: https://huggingface.co/datasets/KaLM-Embedding/KaLM-reranker-training-data.relevance-reranker-v3reranker-scoresruri-v3-dataset-rerankerCreated from hpprc/reranker-scores.
We found that cleaning up noisy positives and negatives in our existing dataset using rerankers' scores had a massive impact on performance.
Concretely:
We averaged the scores from five off‑the‑shelf reranker models.
For "positive" examples (documents that contain the answer string for a given query), we only kept those with an average score ≥ 0.3.
For "negative" examples (documents that do not contain the answer string), we only kept those with an average… See the full description on the dataset page: https://huggingface.co/datasets/cl-nagoya/ruri-v3-dataset-reranker.hpprc_emb_reranker_score
⚠️ お知らせ
よりスコア付したデータ件数とrerankerのバリエーションを増やしたデータセットのhotchpotch/hpprc_emb-scoresも公開しています。
hpprc/emb (便利なデータセットの公開、ありがとうございます)の collection と dataset がペアになっているデータに対し、negative を最大32個ランダムサンプリングしたものを、hotchpotch/japanese-bge-reranker-v2-m3-v1でスコア付けしたものです。
ライセンスは、subset ごとに hpprc/emb に記載のライセンスと同等とします。
スコア作成タイミングの revision に対してスコアを付与しているため、revision を変えると場合によって行ズレやデータ構造の変化が発生する可能性があることに注意が必要です。
例
from datasets import load_dataset
# targets = ("auto-wiki-qa", "4feb2e2492")… See the full description on the dataset page: https://huggingface.co/datasets/hotchpotch/hpprc_emb_reranker_score.mmarco-hard-negatives-reranker-score
hotchpotch/mmarco-hard-negatives-reranker-score
This repository contains data from mMARCO scored using the reranker BAAI/bge-reranker-v2-m3.
Languages Covered
target_languages = [
"english",
"chinese",
"french",
"german",
"indonesian",
"italian",
"portuguese",
"russian",
"spanish",
"arabic",
"dutch",
"hindi",
"japanese",
"vietnamese"
]
Hard Negative Data
The hard negative data is derived from… See the full description on the dataset page: https://huggingface.co/datasets/hotchpotch/mmarco-hard-negatives-reranker-score.japanese-reranker-v2-hard-negatives-scores
hotchpotch/japanese-reranker-v2-hard-negatives-scores
This dataset was used to train the Japanese Reranker v2 family. It was not generated by Japanese Reranker v2 models.
Unified hard-negative score rows generated from the training sources used for Japanese Reranker v2-family experiments.
This dataset keeps teacher scores as floating-point soft labels. It follows the HPPRC Emb Score style: score/id rows and referenced document text collections are stored separately. Each score… See the full description on the dataset page: https://huggingface.co/datasets/hotchpotch/japanese-reranker-v2-hard-negatives-scores.reranker_continuous_filt_max7_train
Reranker training data
This data was generated using 4 steps:
We gathered queries and corresponding text data from 35 high quality datasets covering more than 95 languages.
For datasets which did not already have negative texts for queries, we mined hard negatives using the BAAI/bge-m3 embedding model.
For each query, we selected one positive and one negative text and used Qwen/Qwen2.5-32B-Instruct-GPTQ-Int4 to rate the relatedness of each query-text pair using a token "1", "2"… See the full description on the dataset page: https://huggingface.co/datasets/lightblue/reranker_continuous_filt_max7_train.malaysian-rerankermalaysian-reranker-v2default-680K-bge-reranker-v2-gemmaomnimcp_agentops_vector_reranker_teaser
🔬 INSPECT THE DEEPSEEK-R1 REASONING CHAIN LIVE:
Zero hallucinations. Null syntax errors. 100% AST compiler validated.🌐 Live Interactive Reasoning & Code Inspector: https://emgena.com/trainingslager🎁 Claim your Free Starter Kit (Code: STARTER100): https://emgena.com/trainingslager🏷️ Launch Discount: Get 20 € OFF any 500-incident production suite with code LAUNCH20!
📜 Enterprise Compliance: EU AI Act Articles 50 & 53 certified • 100% DSGVO / GDPR clean • Commercial EULA… See the full description on the dataset page: https://huggingface.co/datasets/emgena/omnimcp_agentops_vector_reranker_teaser.ruri-v3-dataset-reranker-previewcreated from hotchpotch/hpprc_emb-scores
License Information
Name
Type
License (根拠)
Mr. TyDi
Retrieval
Apache 2.0
MIRACL
Retrieval
Apache 2.0
Auto Wiki QA Nemotron
QA & Retrieval
CC-BY-SA-4.0
JSQuAD
QA & Retrieval
CC-BY-SA-4.0
JQaRA (dev, unused)
QA
CC-BY-SA-4.0
JaQuAD
QA & Retrieval
CC-BY-SA-3.0
MKQA
QA
CC-BY-3.0
Quiz Works
QA
Free (二次利用自由)
Quiz No Mori
QA
Free (二次利用自由)
ru-reranker-teacher-scores
Teacher score cache for Russian reranker distillation
Raw logits of BAAI/bge-reranker-v2-m3 for ~5.4M (query, passage) pairs, grouped
by query (1 "pos" slot + candidates), produced for the ru-reranker-edge project.
teacher_scores.jsonl — one line per group: {"qid", "pos_score", "cand_scores": [...]} (raw logits, no normalization).
groups_pool_v5.jsonl — the matching texts: {"qid", "query", "pos_id", "pos_text", "cand_ids", "cand_texts"}.
Sources of queries/candidates:… See the full description on the dataset page: https://huggingface.co/datasets/sshalimov04/ru-reranker-teacher-scores.bge-reranker-dataThe dataset used to train bge-reranker-large and bge-reranker-base.
You can find the fine-tuning script in FlagEmbedding
Citation
If you find this repository useful, please consider giving a star :star: and citation
@misc{bge_embedding,
title={C-Pack: Packaged Resources To Advance General Chinese Embedding},
author={Shitao Xiao and Zheng Liu and Peitian Zhang and Niklas Muennighoff},
year={2023},
eprint={2309.07597},
archivePrefix={arXiv}… See the full description on the dataset page: https://huggingface.co/datasets/Shitao/bge-reranker-data.msmarco-Qwen3-Reranker-0.6Bbge-reranker-qwen3-embedding
bge-reranker → Qwen3-Embedding (0.6B) embedding dataset
Precomputed Qwen3-Embedding-0.6B embeddings for the
Shitao/bge-reranker-data
dataset, together with a deduplicated, index-based reranking layout
(anchor / positive / negative triples) suitable for training a metric-learning
projection head.
The point of this dataset: rank documents with a dot product against cached
vectors — no cross-encoder, no re-reading documents at query time.
What's inside
combined/… See the full description on the dataset page: https://huggingface.co/datasets/npc0/bge-reranker-qwen3-embedding.govie-office-holder-reranker-bilingual-v2
gov.ie Office Holder Reranker Bilingual v2
Bilingual (query, candidate_page) reranking dataset for current Irish government office-holder lookup on public gov.ie pages.
This is a derivative of temsa/govie-office-holder-reranker-dataset-v1 with one important change: every English query is paired with an Irish Gaelic query variant while keeping the same candidate pool and labels.
What changed versus v1
same candidate pages and labels
same office-holder snapshot and… See the full description on the dataset page: https://huggingface.co/datasets/temsa/govie-office-holder-reranker-bilingual-v2.msmarco-Qwen3-Reranker-0.6B-frenchreranker_fulltrain_scored_filteredgovie-dsp-rates-reranker-bilingual-v1
gov.ie DSP Rates Reranker Bilingual v1
Bilingual (query, candidate_page) reranking dataset for gov.ie Department of Social Protection payment-rate lookup.
This release is built from a curated DSP scheme catalog grounded in public gov.ie service pages and the official SW19 Rates of Payment 2026 booklet.
What is in the dataset
English and Irish Gaelic query variants for DSP allowance, benefit and grant lookup
exact-name and rate-lookup queries
page-level positives with… See the full description on the dataset page: https://huggingface.co/datasets/temsa/govie-dsp-rates-reranker-bilingual-v1.VLSP-Legal-Reranker-Train
VLSP Vietnamese Legal Reranker — Training + Eval data
Bộ dữ liệu listwise reranking cho truy hồi điều luật tiếng Việt (VLSP). Đây chính xác là
dữ liệu đã dùng để fine-tune 2 model reranker:
🤗 hoailebads/Qwen3-Reranker-8B-VLSP-Legal-LoRA (bản chính, vượt SOTA VLSP)
🤗 hoailebads/Qwen3-Reranker-0.6B-VLSP-Legal-LoRA (bản nhẹ)
📦 Code, pipeline & script tạo dữ liệu: https://github.com/hoaileba/Qwen-Retrieval-Tuning
(file gốc: dataset/train_rankllama_hn23_v3.jsonl, tạo bằng… See the full description on the dataset page: https://huggingface.co/datasets/hoailebads/VLSP-Legal-Reranker-Train.rlhn-680K-qwen3-reranker-8b
RLHN-680K — Qwen3-Reranker-8B teacher scores (top-200)
The distillation training set used by Tevatron 3.0's listwise-KL reranker
distillation. Each RLHN-680K
query is paired with its candidate passages and a per-passage teacher
relevance score from Qwen3-Reranker-8B
— the soft targets the distillation runs consume.
Schema
Field
Type
Description
query_id
string
RLHN query id
query
string
query text
passages
list of {{title, text}}
candidate passages… See the full description on the dataset page: https://huggingface.co/datasets/utahnlp/rlhn-680K-qwen3-reranker-8b.ssf-dataset-synthetic-reranker-v0
Dataset Card for ssf-dataset-synthetic-reranker-v0
This dataset has been created with distilabel.
Dataset Summary
This dataset contains a pipeline.yaml which can be used to reproduce the pipeline that generated it in distilabel using the distilabel CLI:
distilabel pipeline run --config "https://huggingface.co/datasets/dnth/ssf-dataset-synthetic-reranker-v0/raw/main/pipeline.yaml"
or explore the configuration:
distilabel pipeline info --config… See the full description on the dataset page: https://huggingface.co/datasets/dnth/ssf-dataset-synthetic-reranker-v0.finephrase-gov-reranker-dataset-v2-200
Finephrase Gov Reranker Dataset v2 (200 rows)
Synthetic reranking dataset built from HuggingFaceFW/finephrase FAQ-style questions filtered to Irish public-service domains:
gov.ie
citizensinformation.ie
hse.ie
oireachtas.ie
revenue.ie
Each grouped row contains a query plus one relevant page snippet and a set of hard negatives from the same filtered corpus.
Included Files
train.jsonl, valid.jsonl, test.jsonl
grouped reranker evaluation/training format… See the full description on the dataset page: https://huggingface.co/datasets/temsa/finephrase-gov-reranker-dataset-v2-200.msmarco-Qwen3-Reranker-0.6B-dutch