datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
hh-rlhf
Dataset Card for HH-RLHF
Dataset Summary
This repository provides access to two different kinds of data:
Human preference data about helpfulness and harmlessness from Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. These data are meant to train preference (or reward) models for subsequent RLHF training. These data are not meant for supervised training of dialogue agents. Training dialogue agents on these data is likely… See the full description on the dataset page: https://huggingface.co/datasets/Anthropic/hh-rlhf.hh-rlhf-49k-ja-single-turnThis dataset was created by automatically translating part of "Anthropic/hh-rlhf" into Japanese, and selected for single turn conversations.You can use this dataset for RLHF and DPO.
hh-rlhf repository
https://github.com/anthropics/hh-rlhf
Anthropic/hh-rlhf
https://huggingface.co/datasets/Anthropic/hh-rlhf
full-hh-rlhf
Dataset Card for "full-hh-rlhf"
Anthropic's HH dataset reformatted into prompt, chosen, rejected samples.
wiki-linguarlhflow_mixture_with_math_del_systemRLHF-V-Dataset
Dataset Card for RLHF-V-Dataset
Project Page | Paper | GitHub
Updates
[2024.05.28] 📃 Our RLAIF-V paper is accesible at arxiv now!
[2024.05.20] 🎉 We release a new feedback dataset, RLAIF-V-Dataset, which is a large-scale diverse-task multimodal feedback dataset constructed using open-source models. You can download the corresponding dataset and models (7B, 12B) now!
[2024.04.11] 🔥 Our data is used in MiniCPM-V 2.0, an end-side multimodal large language model that… See the full description on the dataset page: https://huggingface.co/datasets/openbmb/RLHF-V-Dataset.rlhflow_mix_w_aimo_mathtldr-thematicHH-RLHF-Helpful-standardWe process the helpful subset of Anthropic-HH into the standard format. The filtering script is as follows.
def filter_example(example):
if len(example['chosen']) != len(example['rejected']):
return False
if len(example['chosen']) % 2 != 0:
return False
n_rounds = len(example['chosen'])
for i in range(len(example['chosen'])):
if example['chosen'][i]['role'] != ['user', 'assistant'][i % 2]:
return False
if… See the full description on the dataset page: https://huggingface.co/datasets/RLHFlow/HH-RLHF-Helpful-standard.hh-rlhf-helpful-base-trl-style
TRL's Anthropic HH Dataset
We preprocess the dataset using our standard prompt, chosen, rejected format.
Reproduce this dataset
Download the anthropic_hh.py from the https://huggingface.co/datasets/trl-internal-testing/hh-rlhf-helpful-base-trl-style/tree/0.1.0.
Run python examples/datasets/anthropic_hh.py --push_to_hub --hf_entity trl-internal-testing
UltraFeedback-preference-standardWe include all the possible comparisons following the Instruct-GPT. We use the fine-grained_score.
import os
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from datasets import load_dataset, DatasetDict
from transformers import AutoTokenizer
from tqdm import tqdm
from transformers import AutoTokenizer
ds = load_dataset("openbmb/UltraFeedback", split="train")
import itertools
data = []
for example in ds:
prompt = example['instruction']
responses = {}… See the full description on the dataset page: https://huggingface.co/datasets/RLHFlow/UltraFeedback-preference-standard.rlhflow_mix_w_dart_mathrlhflow_mix_del_system_and_empty_round2MM-RLHF
[📖 arXiv Paper]
[📊 Training Code]
[📝 Homepage]
[🏆 Reward Model]
[🔮 MM-RewardBench]
[🔮 MM-SafetyBench]
[📈 Evaluation Suite]
The Next Step Forward in Multimodal LLM Alignment
[2025/02/10] 🔥 We are proud to open-source MM-RLHF, a comprehensive project for aligning Multimodal Large Language Models (MLLMs) with human preferences. This release includes:
A high-quality MLLM alignment dataset.
A strong Critique-Based MLLM reward model and its training algorithm.
A… See the full description on the dataset page: https://huggingface.co/datasets/yifanzhang114/MM-RLHF.rlhflow_mix_del_system_and_empty_roundsocial-reasoning-rlhf-ULTRAFEEDBACK-honesty
Dataset Card for "social-reasoning-rlhf-ULTRAFEEDBACK-honesty"
More Information needed
hh-rlhfThis dataset is part of the Anthropic's HH data used to train their RLHF Assistant https://github.com/anthropics/hh-rlhf.
The data contains the first utterance from human to the dialog agent and the number of words in that utterance. The sampled version is a random sample of size 200.
hh-rlhf-helpful-base
HH-RLHF-Helpful-Base Dataset
Summary
The HH-RLHF-Helpful-Base dataset is a processed version of Anthropic's HH-RLHF dataset, specifically curated to train models using the TRL library for preference learning and alignment tasks. It contains pairs of text samples, each labeled as either "chosen" or "rejected," based on human preferences regarding the helpfulness of the responses. This dataset enables models to learn human preferences in generating helpful responses… See the full description on the dataset page: https://huggingface.co/datasets/trl-lib/hh-rlhf-helpful-base.eli5_rlhfELI5 paired
This is a processed version of the eli5 dataset. The dataset was created following very closely the steps in the stack-exchange-paired dataset. The following steps were applied:
Create pairs (response_j, response_k) where j was rated better than k
Sample at most 10 pairs per question
Shuffle the dataset globally
This dataset is designed to be used for preference learning using techniques such as Reinforcement Learning from Human Feedback. The processing notebook is in the… See the full description on the dataset page: https://huggingface.co/datasets/vincentmin/eli5_rlhf.oasst1_pairwise_rlhf_reward
Dataset Card for "oasst1_pairwise_rlhf_reward"
OASST1 dataset preprocessed for reward modeling:
import pandas as pd
from datasets import load_dataset,concatenate_datasets, Dataset, DatasetDict
import numpy as np
dataset = load_dataset("OpenAssistant/oasst1")
df=concatenate_datasets(list(dataset.values())).to_pandas()
m2t=df.set_index("message_id")['text'].to_dict()
m2r=df.set_index("message_id")['role'].to_dict()
m2p=df.set_index('message_id')['parent_id'].to_dict()… See the full description on the dataset page: https://huggingface.co/datasets/tasksource/oasst1_pairwise_rlhf_reward.Mistral-PRM-DataSee https://github.com/RLHFlow/RLHF-Reward-Modeling/tree/main/math-rm for more data information.
social-reasoning-rlhf
Dataset Summary
This repository provides access to a social reasoning dataset that aims to provide signal to how humans navigate social situations, how they reason about them and how they understand each other. It contains questions probing people's thinking and understanding of various social situations.
This dataset was created by collating a set of questions within the following social reasoning tasks:
understanding of emotions
intent recognition
social norms
social… See the full description on the dataset page: https://huggingface.co/datasets/ProlificAI/social-reasoning-rlhf.Nemotron-RLHF-GenRM-v1
Dataset Description:
This dataset is designed to train Generative Reward Models (GenRMs). It leverages reinforcement learning at scale to train accurate and robust GenRMs that generalize better than traditional Bradley-Terry models and reduce the risk of reward hacking.
The dataset is composed of:
Preference data focused on diverse domains
A synthetic safety blend
The data follows a "meta-prompt" structure where the model is instructed to act as an expert evaluation judge. For… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/Nemotron-RLHF-GenRM-v1.Helpsteer-preference-standardhh-rlhf_generation_rewards_iter2zhihu_rlhf_3khh_rlhf_cn
hh-rlhf中文翻译版本
基于Anthropic论文Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback 开源的helpful 和harmless数据，使用翻译工具进行了翻译。hh_rlhf_train.jsonl 合并中英文训练集数据 清洗过后17万条hh_rlhf_test.jsonl 合并中英文测试集数据 清洗过后9千条harmless_base_cn_train.jsonl 42394条harmless_base_cn_test.jsonl 2304条helpful_base_cn_train.jsonl 43722条helpful_base_cn_test.jsonl 2346条
实验报告
相关rlhf实验报告:https://zhuanlan.zhihu.com/p/652044120
proofwriter
ProofWriter — structured
A cleaned, structured build of AI2's ProofWriter
for logical entailment with reasoning-graph supervision. Each row is one theory
(facts + Horn-clause rules) with the questions posed against it; the zip's formal string
reps are parsed into typed atoms (subject, relation, object, polarity), and every
question keeps its gold answer and gold derivation as a structured proof graph — so no
natural-language reverse-engineering is needed downstream.… See the full description on the dataset page: https://huggingface.co/datasets/rlhf-and-friends/proofwriter.hh-rlhf-trl-style
TRL's Anthropic HH Dataset
We preprocess the dataset using our standard prompt, chosen, rejected format.
Reproduce this dataset
Download the anthropic_hh.py from the https://huggingface.co/datasets/trl-internal-testing/hh-rlhf-trl-style/tree/0.1.0.
Run python examples/datasets/anthropic_hh.py --push_to_hub --hf_entity trl-internal-testing
RLHFlow-SFT-Dataset-ver2