datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
ultrafeedback-binarized-preferences-cleaned
UltraFeedback - Binarized using the Average of Preference Ratings (Cleaned)
This dataset represents a new iteration on top of argilla/ultrafeedback-binarized-preferences,
and is the recommended and preferred dataset by Argilla to use from now on when fine-tuning on UltraFeedback.
Read more about Argilla's approach towards UltraFeedback binarization at argilla/ultrafeedback-binarized-preferences/README.md.
Differences with argilla/ultrafeedback-binarized-preferences… See the full description on the dataset page: https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences-cleaned.Capybara-Preferences
Dataset Card for Capybara-Preferences
This dataset has been created with distilabel.
Dataset Summary
This dataset is built on top of LDJnr/Capybara, in order to generate a preference
dataset out of an instruction-following dataset. This is done by keeping the conversations in the column conversation but splitting
the last assistant turn from it, so that the conversation contains all the turns up until the last user's turn, so that it can be reused… See the full description on the dataset page: https://huggingface.co/datasets/argilla/Capybara-Preferences.dialect-preferences
DiaLLM — Pooled Preference Dataset (Implicit Thread)
Part of DiaLLM: An Investigation into the Robustness-Generation Gap in
English Dialect Adaptation (EMNLP 2026 Main).
45,690 preference pairs, pooling all three variety-specific sets
(Australian,
Northern British,
Indian) without
variety targeting. Used for implicit-thread DPO training, where the three
varieties are pooled rather than targeted individually, preserving the
variety-agnostic objective of that thread.… See the full description on the dataset page: https://huggingface.co/datasets/surrey-nlp/dialect-preferences.ultrafeedback-multi-binarized-preferences-cleaned
UltraFeedback - Multi-Binarized using the Average of Preference Ratings (Cleaned)
This dataset represents a new iteration on top of argilla/ultrafeedback-binarized-preferences-cleaned,
and has been created to explore whether DPO fine-tuning with more than one rejection per chosen response helps the model perform better in the
AlpacaEval, MT-Bench, and LM Eval Harness benchmarks.
Read more about Argilla's approach towards UltraFeedback binarization at… See the full description on the dataset page: https://huggingface.co/datasets/argilla/ultrafeedback-multi-binarized-preferences-cleaned.gigaverbo-v2-preferences
GigaVerbo-v2 Preferences: A Hybrid-Reasoning Portuguese Preference Dataset
Dataset Summary
GigaVerbo-v2 Preferences is a preference dataset designed for Direct Preference Optimization (DPO) and other direct alignment algorithms. The dataset comprises approximately 27.8 million tokens across 28,437 preference pairs, organized into 4 distinct subsets covering both quality-focused and safety-focused alignment. It is entirely composed of high-quality, LLM-generated data… See the full description on the dataset page: https://huggingface.co/datasets/Polygl0t/gigaverbo-v2-preferences.Capybara-Preferences-Filtered
Dataset Card for Capybara-Preferences-Filtered
This dataset has been created with distilabel, plus some extra post-processing steps described below.
Dataset Summary
This dataset is built on top of argilla/Capybara-Preferences, but applies a further in detail filtering.
The filtering approach has been proposed and shared by @LDJnr, and applies the following:
Remove responses from the assistant, not only in the last turn, but also in intermediate… See the full description on the dataset page: https://huggingface.co/datasets/argilla/Capybara-Preferences-Filtered.ultrafeedback-binarized-preferences-cleaned
UltraFeedback - Binarized using the Average of Preference Ratings (Cleaned)
This dataset represents a new iteration on top of argilla/ultrafeedback-binarized-preferences,
and is the recommended and preferred dataset by Argilla to use from now on when fine-tuning on UltraFeedback.
Read more about Argilla's approach towards UltraFeedback binarization at argilla/ultrafeedback-binarized-preferences/README.md.
Differences with argilla/ultrafeedback-binarized-preferences… See the full description on the dataset page: https://huggingface.co/datasets/pharaouk/ultrafeedback-binarized-preferences-cleaned.DiscoverLLM-multiturn-preferences
DiscoverLLM: Multi-turn Preference Dataset
Multi-turn dialogue data with scored candidate completions, produced by best-of-N
synthesis over the DiscoverLLM user simulator
(paper · project page).
Each example is a single turn of a simulated user–assistant conversation with one of
several candidate assistant responses and an associated reward score, intended for
offline DPO / GRPO / reward-model training.
Configs
Config
Rows
Task
creative_writing
3,052… See the full description on the dataset page: https://huggingface.co/datasets/kixlab/DiscoverLLM-multiturn-preferences.innoduel-rlhf-real-world-human-preferences-sample
Real-World Human Pairwise Preferences — Public Sample
📦 This is a free, public sample of a commercial dataset.
It contains 1,350 rows curated for inspection. The full dataset has 1.5 million
human pairwise-preference decisions.
Full dataset: https://huggingface.co/datasets/NordosoftOy/innoduel-rlhf
Request access / licensing: see § Access to the full dataset — contact kari.nieminen@nordo.fi.
Use this sample to evaluate the data's quality, structure and… See the full description on the dataset page: https://huggingface.co/datasets/NordosoftOy/innoduel-rlhf-real-world-human-preferences-sample.paraphrasing-preferences-orpo-dpo
Paraphrasing Preference Dataset
A preference dataset for training paraphrase models via DPO, RLHF, or ORPO. Each example contains a source text, a task-specific prompt, and a chosen/rejected paraphrase pair ranked by a composite quality score.
Dataset Summary
Train
Val
Total
Examples
852
95
947
Sources: Quora questions (571), SQuAD 2.0 sentences (218), CNN News sentences (158). The val split is stratified by excellent_in, category, and binned total_delta.… See the full description on the dataset page: https://huggingface.co/datasets/alecccdd/paraphrasing-preferences-orpo-dpo.ultrafeedback-binarized-preferences-cleaned-no-refusals
UltraFeedback Binarized Preferences Cleaned No Refusals
A Minos-cleaned version of argilla/ultrafeedback-binarized-preferences-cleaned for use as a neutral helpfulness DPO anchor. Rows are removed when either the chosen or rejected assistant response is classified as a refusal by NousResearch/Minos-v1.
Cleaning version: minos-only-v1-2026-06-23
See manifest.json in the repository files for counts and endpoint metadata.
lima-qwen2.5-7b-pairrm-preferences
LIMA × Qwen2.5-7B-Instruct × PairRM preference dataset
Preference dataset built for Assignment 4 of the alignment course.
How it was built
Source instructions: 50 instructions sampled with seed=42 from the GAIR/lima training split.
Candidate generation: For each instruction we sampled 5 responses from Qwen/Qwen2.5-7B-Instruct using the official chat template (temperature=0.9, top_p=0.95, max_new_tokens=512).
Ranking: All 5 candidates per instruction were ranked with… See the full description on the dataset page: https://huggingface.co/datasets/Barryzbr12/lima-qwen2.5-7b-pairrm-preferences.assignment4-pairrm-preferences-submit