datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
RLHF-V-Dataset
Dataset Card for RLHF-V-Dataset
Project Page | Paper | GitHub
Updates
[2024.05.28] 📃 Our RLAIF-V paper is accesible at arxiv now!
[2024.05.20] 🎉 We release a new feedback dataset, RLAIF-V-Dataset, which is a large-scale diverse-task multimodal feedback dataset constructed using open-source models. You can download the corresponding dataset and models (7B, 12B) now!
[2024.04.11] 🔥 Our data is used in MiniCPM-V 2.0, an end-side multimodal large language model that… See the full description on the dataset page: https://huggingface.co/datasets/openbmb/RLHF-V-Dataset.eli5_rlhfELI5 paired
This is a processed version of the eli5 dataset. The dataset was created following very closely the steps in the stack-exchange-paired dataset. The following steps were applied:
Create pairs (response_j, response_k) where j was rated better than k
Sample at most 10 pairs per question
Shuffle the dataset globally
This dataset is designed to be used for preference learning using techniques such as Reinforcement Learning from Human Feedback. The processing notebook is in the… See the full description on the dataset page: https://huggingface.co/datasets/vincentmin/eli5_rlhf.Nemotron-RLHF-GenRM-v1
Dataset Description:
This dataset is designed to train Generative Reward Models (GenRMs). It leverages reinforcement learning at scale to train accurate and robust GenRMs that generalize better than traditional Bradley-Terry models and reduce the risk of reward hacking.
The dataset is composed of:
Preference data focused on diverse domains
A synthetic safety blend
The data follows a "meta-prompt" structure where the model is instructed to act as an expert evaluation judge. For… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/Nemotron-RLHF-GenRM-v1.social-reasoning-rlhf
Dataset Summary
This repository provides access to a social reasoning dataset that aims to provide signal to how humans navigate social situations, how they reason about them and how they understand each other. It contains questions probing people's thinking and understanding of various social situations.
This dataset was created by collating a set of questions within the following social reasoning tasks:
understanding of emotions
intent recognition
social norms
social… See the full description on the dataset page: https://huggingface.co/datasets/ProlificAI/social-reasoning-rlhf.eli5_rlhf_explainlikeim5
ELI5 paired
This is a processed version of the eli5 dataset.
Compared to "eli5_rlhf", this dataset contains only QA pairs from the train split of the eli5 dataset and only from the subreddit explainlikeimfive.
Furthermore, the function
def get_question(example):
title = example["title"]
selftext = example["selftext"]
if selftext:
if selftext[-1] not in [".", "?", "!"]:
seperator = ". "
else:
seperator = " "
question = title… See the full description on the dataset page: https://huggingface.co/datasets/vincentmin/eli5_rlhf_explainlikeim5.gsm8k-qwen3-0.6B-rollouts
GSM8K Qwen3-0.6B Rollouts
Verifier-labeled solutions sampled from Qwen/Qwen3-0.6B with vLLM
0.26.0 and its pytorch top-k/top-p
sampler for every prompt in the train and test splits of
openai/gsm8k.
Dataset size and label balance
train: 7,473 prompt rows; 747,300 rollouts; 602,611 correct (80.64%), 144,689 incorrect (19.36%)
test: 1,319 prompt rows; 131,900 rollouts; 102,245 correct (77.52%), 29,655 incorrect (22.48%)
Each row contains one source prompt and 100… See the full description on the dataset page: https://huggingface.co/datasets/RLHF-Book/gsm8k-qwen3-0.6B-rollouts.RLHF-VBorrowed from: https://huggingface.co/datasets/openbmb/RLHF-V-Dataset
You can use it in LLaMA Factory by specifying dataset: rlhf_v.
HH-RLHF-MetaAligner-Data
Dataset Card for HH-RLHF-MetaAligner-Data
Dataset Summary
This repository provides access to 2 subsets of the MetaAligner data built from HH-RLHF:
The equal-preference subset is included in: equal_train.json and equal_val.json.
The preference subset is included in: preference_train.json, preference_val.json, and preference_test.json.
Disclaimer: As the dialogue data comes from HH-RLHF, the data contain content that may be offensive or upsetting. Topics include, but are… See the full description on the dataset page: https://huggingface.co/datasets/MetaAligner/HH-RLHF-MetaAligner-Data.korean_rlhf_dataset성균관대학교 산학협력프로젝트 과정에서 한국어 llm 모델 SFT 학습을 위해 구축한 데이터셋 입니다.2023-09-25오픈 어시스턴트 data에서 오픈 어시스턴트를 포함하는 데이터 삭제-> 답변에 오픈 어시스턴트라고 하는 경우가 나오기 때문또한 스탠포드 대학 번역 데이터에서 번역 과정 오류로 input에 입력없음 과 같이 추가된 부분 삭제그리고 <unk> 등으로 gpt 상에서 번역 오류가 난 것들을 삭제
자연스러움을 위해 stanford alpaca data, oig_chip2를 ChatGPT3.5 turbo 16k를 이용하여 새롭게 전처리 과정을 거쳤습니다.https://github.com/JoJo0217/rlhf_korean_dataset/tree/main여기에서 자세한 설명을 볼 수 있으며데이터의 구성은 다음과 같습니다.
데이터 구성
데이터 종류
개수
url
koalpaca v1.1
21155… See the full description on the dataset page: https://huggingface.co/datasets/jojo0217/korean_rlhf_dataset.hh-rlhf-formatted
Reformatted version of Anthropic's hh-rlhf dataset
Original available at https://huggingface.co/datasets/Anthropic/hh-rlhf. (Does not include red teaming data)
RLHF datasets are in general defined as a collection of triples D={(x,y_1,y_2)_n} where x is the prompt, y_1 the chosen reponse and y_2 the rejected response.
The original dataset provides two columns, "chosen"=x+y_1 and "rejected"=x+y_2.
Various RLHF setups may require either format, so in this dataset we keep the original… See the full description on the dataset page: https://huggingface.co/datasets/KHuss/hh-rlhf-formatted.hh-rlhf-sft-rejected
hh-rlhf-sft-rejected
This is a highly curated, cleaned, and structurally reformatted version of Anthropic's HH-RLHF (Helpful and Harmless) dataset, designed directly for high-quality Supervised Fine-Tuning (SFT).
We have stripped away unnecessary preference learning features to deliver a single, standard instruction-following target derived from the original interactions.
Dataset Enhancements & Modifications
Extraction: Removed the redundant chosen column.… See the full description on the dataset page: https://huggingface.co/datasets/Arko007/hh-rlhf-sft-rejected.hh-rlhf-dpo-native
hh-rlhf-dpo-native
Anthropic/hh-rlhf in the native TRL conversational-preference format, with zero-gradient
and unparsable pairs removed.
The original dataset ships two raw strings (chosen, rejected) containing the entire
conversation serialized with \n\nHuman: / \n\nAssistant: separators. Every user has to
write their own parser, and that parser has to make a judgement call on ~2% of rows that are
corrupted. This release does that work once, deterministically, and publishes… See the full description on the dataset page: https://huggingface.co/datasets/Archangel-system/hh-rlhf-dpo-native.mtmc-rlhf
mtmc-rlhf
The mtmc-rlhf (multi-task multi-capability rlhf) dataset is in Chinese and consists primarily of text prompts submitted to a large language model API, enriched by a small portion of prompts crafted by our annotators.
Each sample in the dataset represents a multi-turn session between a user and the language model with a category label. The final query within the session has several distinct responses as well as their corresponding preference rank sorted by annotators.
The… See the full description on the dataset page: https://huggingface.co/datasets/quanshr/mtmc-rlhf.korean_rlhf_content_filtered
Korean RLHF Content Filtered
Dataset Summary
This dataset is a cleaned, content-only derivative of:
Source dataset: jojo0217/korean_rlhf_dataset
Source URL: https://huggingface.co/datasets/jojo0217/korean_rlhf_dataset
Each row has a single content field suitable for LM pretraining/SFT-style text modeling.
Construction
Content construction rule
For each source row:
If input is empty: content = instruction + "\n" + output
If input is not empty:… See the full description on the dataset page: https://huggingface.co/datasets/alwaysgood/korean_rlhf_content_filtered.cs217-rlhf-dataset
CS217 Fixed HH-RLHF Dataset
Fixed subset of Anthropic's HH-RLHF dataset for reproducible RLHF experiments
Created for Stanford CS217: Hardware Accelerators for Machine Learning - Final Project
🔗 GitHub Repository: CS217-Final-Project
Dataset Description
This is a fixed subset of the Anthropic/hh-rlhf dataset, created to ensure reproducible experiments across all runs. The dataset contains human preference pairs for training reward models and RLHF (Reinforcement Learning… See the full description on the dataset page: https://huggingface.co/datasets/hivamoh/cs217-rlhf-dataset.hh-rlhf-helpful-dpo-10k
HH-RLHF Helpful DPO Preference Pairs · 10k
10,000 real human preference pairs for teaching a tiny language model (≤50M params)
what a good assistant sounds like — more helpful, more natural, less evasive.
Why this dataset exists
This is the preference-tuning stage of an end-to-end tiny-model training pipeline:
Pretraining ──► SFT ──► DPO (this dataset) ──► Tiny Edge Assistant
After SFT teaches the model how to speak, this dataset… See the full description on the dataset page: https://huggingface.co/datasets/salisai/hh-rlhf-helpful-dpo-10k.hh-rlhf-sycophantic
hh-rlhf-sycophantic
A re-judged subset of Anthropic/hh-rlhf
(helpful-online config) where each preference pair has a syc_score ∈ [-4, +4]
indicating how much MORE sycophantic the chosen response is than the rejected
one, per an LLM judge with positional-bias debiasing.
DPO-training on syc_score > threshold pairs is a knob to amplify sycophancy
in a target model (reproducing Sharma et al. 2023, "Towards Understanding Sycophancy
in Language Models" §3.3 finding that PM
training data… See the full description on the dataset page: https://huggingface.co/datasets/wassname/hh-rlhf-sycophantic.pandora-rlhf
Pandora RLHF
A Reinforcement Learning from Human Feedback (RLHF) dataset for Direct Preference Optimization (DPO) fine-tuning of the Pandora Large Language Model (LLM).
The dataset is based on the anthropic/hh-rlhf dataset.
Copyright and license
Copyright (c) 2024, Danilo Peixoto Ferreira. All rights reserved.
Project developed under a BSD-3-Clause license.
dolly-15k-rlhf-instructgpt-format
Dolly 15k RLHF Datasets in InstructGPT Format
This repository packages databricks/databricks-dolly-15k into three RLHF-oriented
dataset configurations inspired by the InstructGPT data flow:
sft: supervised fine-tuning examples with prompt, completion, and text.
rm_schema: reward-modeling schema/prompt pool with empty chosen and rejected
fields, reference_response, and ready_for_rm=false.
rm_synthetic: reward-modeling proxy pairs where Dolly reference_response is
used as chosen and… See the full description on the dataset page: https://huggingface.co/datasets/costadev00/dolly-15k-rlhf-instructgpt-format.HH-RLHF-Cleaned-GRPO
HH-RLHF-Cleaned-GRPO
Deep-cleaned for GRPO/RL training | 1,847 examples | 187 bugs fixed
📋 Dataset Description
Safety and helpfulness prompts extracted from Anthropic's HH-RLHF dataset. This cleaned version removes 100 duplicate prompts and 87 whitespace issues that would waste training compute on redundant data.
Original source: Anthropic/hh-rlhf by Anthropic
📊 Cleaning Statistics
Metric
Value
Original examples
1,947
Clean… See the full description on the dataset page: https://huggingface.co/datasets/Eyght/HH-RLHF-Cleaned-GRPO.hh-rlhf-chat-templateolmo-2-1124-7b-four-stage-samples-rlhf-diversity
OLMo-2-1124-7B four-stage samples (for diversity research)
This dataset contains K=10 sampled responses per prompt from four stages
of the OLMo-2-1124-7B post-training pipeline on two prompt sets, produced
for the ICL diversity metric paper (Khoriaty, Williams-King, Feng, in preparation).
Why this exists. As of April 2026, no prior work releases ≥10 samples
per prompt from a paired SFT/DPO/RL pipeline. Kirk et al.'s original
RLHF-gen-div paper logged their K=16 BoN outputs to a… See the full description on the dataset page: https://huggingface.co/datasets/AMindToThink/olmo-2-1124-7b-four-stage-samples-rlhf-diversity.innoduel-rlhf-real-world-human-preferences-sample
Real-World Human Pairwise Preferences — Public Sample
📦 This is a free, public sample of a commercial dataset.
It contains 1,350 rows curated for inspection. The full dataset has 1.5 million
human pairwise-preference decisions.
Full dataset: https://huggingface.co/datasets/NordosoftOy/innoduel-rlhf
Request access / licensing: see § Access to the full dataset — contact kari.nieminen@nordo.fi.
Use this sample to evaluate the data's quality, structure and… See the full description on the dataset page: https://huggingface.co/datasets/NordosoftOy/innoduel-rlhf-real-world-human-preferences-sample.cvalues_rlhf_jp_cot
puwaer/cvalues_rlhf_jp_cot
This dataset is based on Skepsun/cvalues_rlhf and has been translated into appropriate Japanese for DPO (Direct Preference Optimization).
For the prompt and rejected (negative example) fields, outputs from huihui-ai/Huihui-gpt-oss-20b-mxfp4-abliterated-v2 were used.
For the chosen (positive example) field, outputs from openai/gpt-oss-20b were used.
For the chosen chain of thought field, outputs from Qwen/Qwen3-235B-A22B-Instruct-2507 were used.
For… See the full description on the dataset page: https://huggingface.co/datasets/puwaer/cvalues_rlhf_jp_cot.WiredBrain-RLHF
WiredBrain-RLHF: Production-Grade Data for High-Integrity AI Alignment
Overview
WiredBrain-RLHF transforms the foundational Anthropic HH-RLHF dataset (148K conversations) into a production-grade training resource through systematic data enrichment. Engineered by Shubham Dev (Department of Computer Science, Jaypee University of Information Technology, India), this dataset powers high-stakes AI alignment research requiring factual integrity and entity preservation.… See the full description on the dataset page: https://huggingface.co/datasets/pheonix-delta/WiredBrain-RLHF.Enterprise-RLHF-Preferences-10k-Sample
🏆 Enterprise RLHF Preference Dataset (10k Sample)
⚠️ RESEARCH & EVALUATION ONLY ⚠️ This is a 10,000-sample preview of the full 60k Enterprise Corpus. For the full commercial license and access to the complete dataset, please contact: [ alinmatei.dev@gmail.com]
📖 Overview
This dataset represents a premium corpus for Reinforcement Learning from Human Feedback (RLHF) and Reward Model (RM) training. Unlike standard web-scraped datasets, this corpus focuses on… See the full description on the dataset page: https://huggingface.co/datasets/hallinh/Enterprise-RLHF-Preferences-10k-Sample.cvalues_rlhf_en
puwaer/cvalues_rlhf_jp
This dataset is based on Skepsun/cvalues_rlhf and has been translated into appropriate English for DPO (Direct Preference Optimization).
For the prompt and rejected (negative example) fields, outputs from huihui-ai/Huihui-gpt-oss-20b-mxfp4-abliterated-v2 were used.
For the chosen (positive example) field, outputs from openai/gpt-oss-20b were used.
このデータセットは、Skepsun/cvalues_rlhfをもとに、適切な英語に翻訳したdpo用のデータセットです。… See the full description on the dataset page: https://huggingface.co/datasets/puwaer/cvalues_rlhf_en.Evol-hh-rlhf-gen3-1k
Evol-hh-rlhf-gen3-1k
Evol-hh-rlhf-gen3-1kは、
kunishou/hh-rlhf-49k-jaをseed tasksとして
Evol-Instructionの手法
mistralai/Mixtral-8x22B-Instruct-v0.1
で作った合成データ(Synthetic data)です。モデルの利用にはDeepinfraを利用しています。
Dataset Details
Dataset Description
Curated by: HachiML
Language(s) (NLP): Japanese
License: Apache 2.0
Github: Evol-Instruct-jp
Uses
# library
fromdatasets import load_dataset
# Load dataset.
dataset =… See the full description on the dataset page: https://huggingface.co/datasets/HachiML/Evol-hh-rlhf-gen3-1k.cvalues_rlhf_jp
puwaer/cvalues_rlhf_jp
This dataset is based on Skepsun/cvalues_rlhf and has been translated into appropriate Japanese for DPO (Direct Preference Optimization).
For the prompt and rejected (negative example) fields, outputs from huihui-ai/Huihui-gpt-oss-20b-mxfp4-abliterated-v2 were used.
For the chosen (positive example) field, outputs from openai/gpt-oss-20b were used.
このデータセットは、Skepsun/cvalues_rlhfをもとに、適切な日本語に翻訳したdpo用のデータセットです。… See the full description on the dataset page: https://huggingface.co/datasets/puwaer/cvalues_rlhf_jp.cvalues_rlhf_en_cot
puwaer/cvalues_rlhf_en_cot
This dataset is based on Skepsun/cvalues_rlhf and has been translated into appropriate English for DPO (Direct Preference Optimization).
For the prompt and rejected (negative example) fields, outputs from huihui-ai/Huihui-gpt-oss-20b-mxfp4-abliterated-v2 were used.
For the chosen (positive example) field, outputs from openai/gpt-oss-20b were used.
For the chosen chain of thought field, outputs from Qwen/Qwen3-235B-A22B-Instruct-2507 were used.
For the… See the full description on the dataset page: https://huggingface.co/datasets/puwaer/cvalues_rlhf_en_cot.