Datasetpublic

salisai/hh-rlhf-helpful-dpo-10k

HH-RLHF Helpful DPO Preference Pairs · 10k 10,000 real human preference pairs for teaching a tiny language model (≤50M params) what a good assistant sounds like — more helpful, more natural, less evasive. Why this dataset exists This is the preference-tuning stage of an end-to-end tiny-model training pipeline: Pretraining ──► SFT ──► DPO (this dataset) ──► Tiny Edge Assistant After SFT teaches the model how to speak, this dataset… See the full description on the dataset page: https://huggingface.co/datasets/salisai/hh-rlhf-helpful-dpo-10k.

sourceHugging Facemitupdated 1mo agoView on Hugging Face

0likes44downloads

Dataset Card

<p align="center"> <img src="https://huggingface.co/datasets/salisai/hh-rlhf-helpful-dpo-10k/resolve/main/logo.svg" alt="TinyML DPO" width="90"/> <h1 align="center">HH-RLHF Helpful DPO Preference Pairs · 10k</h1> <p align="center"> <b>10,000 real human preference pairs</b> for teaching a tiny language model (≤50M params) <i>what a good assistant sounds like</i> — more helpful, more natural, less evasive. </p> <p align="center"> <a href="https://huggingface.co/datasets/Anthropic/hh-rlhf"><img alt="Source" src="https://img.shields.io/badge/Source-Anthropic%2Fhh--rlhf-8A2BE2"></a> <a href="#usage"><img alt="Usage" src="https://img.shields.io/badge/Stage-DPO%20(after%20SFT)-0FB5AE"></a> <a href="#roadmap"><img alt="Vision" src="https://img.shields.io/badge/Target-Edge%20Devices-FF6B6B"></a> </p> </p>

Why this dataset exists

This is the preference-tuning stage of an end-to-end tiny-model training pipeline:

Pretraining ──► SFT ──► DPO (this dataset) ──► Tiny Edge Assistant

After SFT teaches the model how to speak, this dataset teaches it which way of speaking is better. Every row is a real human preference judgment from the Anthropic HH-RLHF collection:

—chosen — the response a good, helpful support assistant should sound like
—rejected — the evasive, unhelpful, or robotic responses to avoid

This is the first step of a longer journey: squeezing as much intelligence as possible out of tiny models that can run on edge devices — phones, microcontrollers, and offline hardware.

Dataset description


Source	`Anthropic/hh-rlhf` — `data_dir="helpful-base"`, `split="train"`
Sampling	`shuffle(seed=42)` → first 10,000 valid pairs
Language	English only
Columns	`prompt`, `chosen`, `rejected`
Format	Parquet (`train.parquet`)

Column layout

Column	Description
`prompt`	Full conversation history up to the final `Assistant:` turn (Human/Assistant dialogue)
`chosen`	The assistant reply a human preferred
`rejected`	The assistant reply a human did not prefer

Corpus statistics

Metric	Prompt	Chosen	Rejected
Avg length (words)	92.9	44.1	34.4
Max length (words)	753	540	292

Usage

python

from datasets import load_dataset
from trl import DPOTrainer

ds = load_dataset("salisai/hh-rlhf-helpful-dpo-10k", split="train")

# rows are already DPO-ready:
# ds[0] = {"prompt": ..., "chosen": ..., "rejected": ...}

You can pass it straight into DPOTrainer with:

python

DPOTrainer(
    model=model,                    # your SFT-finished tiny model (≤50M)
    train_dataset=ds,
    beta=0.1,
    ...
)

Sample row

Prompt

Human: When was Pizza Hut founded?

Assistant: Hm... Pizza Hut was founded in 1958.

Human: What was the founders name?

Chosen (preferred)

Oh sorry, my research doesn't have that. I was just able to get the founder of the company who also opened it.

Rejected (avoid)

Can you tell me what you're looking for? Does "the founder's name" mean that you want to know the name of the founder of the chain of restaurants that operates in many places today? Or maybe you...

Roadmap: tiny models on the edge

Stage	Dataset	Goal
1. Pretraining	token streams	raw language ability
2. SFT	instruction pairs	learn to answer
3. DPO (this)	hh-rlhf helpful pairs	*learn how to answer well*
4. Deployment	—	run on edge devices

Everything here is deliberately small and English-only so a ≤50M-param model can actually absorb the signal. The end goal: useful, natural, non-evasive assistants running on resource-constrained edge hardware — this 10k preference set is the alignment step that makes them sound human instead of robotic.

Source & license

—Source dataset: Anthropic/hh-rlhf — Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback (Bai et al., 2022)
—Built from: the helpful-base subset, human preference labels collected by Anthropic.
—This repository: extracted, cleaned (chosen/rejected prompts validated to match), and formatted into DPO-ready rows.
—License: mit

<div align="center"> <i>Built for a 50M-param experiment — squeezing intelligence, one preference pair at a time.</i> </div>