datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
PKU-SafeRLHF
Dataset Card for PKU-SafeRLHF
Warning: this dataset contains data that may be offensive or harmful. The data are intended for research purposes, especially research that can make models less harmful. The views expressed in the data do not reflect the views of PKU-Alignment Team or any of its members.
[🏠 Homepage] [🤗 Single Dimension Preference Dataset] [🤗 Q-A Dataset] [🤗 Prompt Dataset]
Citation
If PKU-SafeRLHF has contributed to your work, please consider citing… See the full description on the dataset page: https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF.Magpie-Qwen2.5-Pro-1M-v0.1
Project Web: https://magpie-align.github.io/
Arxiv Technical Report: https://arxiv.org/abs/2406.08464
Codes: https://github.com/magpie-align/magpie
Abstract
Click Here
High-quality instruction data is critical for aligning large language models (LLMs). Although some models, such as Llama-3-Instruct, have open weights, their alignment data remain private, which hinders the democratization of AI. High human labor costs and a limited, predefined scope for prompting prevent… See the full description on the dataset page: https://huggingface.co/datasets/Magpie-Align/Magpie-Qwen2.5-Pro-1M-v0.1.indic-align
IndicAlign
A diverse collection of Instruction and Toxic alignment datasets for 14 Indic Languages. The collection comprises of:
IndicAlign - Instruct
Indic-ShareLlama
Dolly-T
OpenAssistant-T
WikiHow
IndoWordNet
Anudesh
Wiki-Conv
Wiki-Chat
IndicAlign - Toxic
HHRLHF-T
Toxic-Matrix
We use IndicTrans2 (Gala et al., 2023) for the translation of the datasets.
We recommend the readers to check out our paper on Arxiv for detailed information on the curation process of these… See the full description on the dataset page: https://huggingface.co/datasets/ai4bharat/indic-align.PKU-SafeRLHF-10K
Paper
You can find more information in our paper.
Dataset Paper: https://arxiv.org/abs/2307.04657
Magpie-Llama-3.1-Pro-300K-Filtered
Project Web: https://magpie-align.github.io/
Arxiv Technical Report: https://arxiv.org/abs/2406.08464
Codes: https://github.com/magpie-align/magpie
Abstract
Click Here
High-quality instruction data is critical for aligning large language models (LLMs). Although some models, such as Llama-3-Instruct, have open weights, their alignment data remain private, which hinders the democratization of AI. High human labor costs and a limited, predefined scope for prompting prevent… See the full description on the dataset page: https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered.PKU-SafeRLHF-30K
Dataset Card for PKU-SafeRLHF
Warning: this dataset contains data that may be offensive or harmful. The data are intended for research purposes, especially research that can make models less harmful. The views expressed in the data do not reflect the views of PKU-Alignment Team or any of its members.
Dataset Summary
The preference dataset consists of 30k+ expert comparison data. Each entry in this dataset includes two responses to a question, along with safety… See the full description on the dataset page: https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF-30K.Magpie-Llama-3.1-Pro-MT-300K-Filtered
Project Web: https://magpie-align.github.io/
Arxiv Technical Report: https://arxiv.org/abs/2406.08464
Codes: https://github.com/magpie-align/magpie
Abstract
Click Here
High-quality instruction data is critical for aligning large language models (LLMs). Although some models, such as Llama-3-Instruct, have open weights, their alignment data remain private, which hinders the democratization of AI. High human labor costs and a limited, predefined scope for prompting prevent… See the full description on the dataset page: https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-MT-300K-Filtered.targeting-alignment
Dataset Card
The datasets in this repository correspond to the embeddings used in "Targeting Alignment: Extracting Safety Classifiers of Aligned LLMs".
For each model, source dataset (input prompts) and setting (benign or adversarial), the corresponding dataset contains the base input prompt, the (deterministic) output of the model, the representations of the input at each layer of the model and the corresponding unsafe/safe labels (1 for unsafe, 0 for safe).
Dataset… See the full description on the dataset page: https://huggingface.co/datasets/jcnf/targeting-alignment.Magpie-Llama-3.3-Pro-1M-v0.1
Project Web: https://magpie-align.github.io/
Arxiv Technical Report: https://arxiv.org/abs/2406.08464
Codes: https://github.com/magpie-align/magpie
Abstract
Click Here
High-quality instruction data is critical for aligning large language models (LLMs). Although some models, such as Llama-3-Instruct, have open weights, their alignment data remain private, which hinders the democratization of AI. High human labor costs and a limited, predefined scope for prompting prevent… See the full description on the dataset page: https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.3-Pro-1M-v0.1.Magpie-Llama-3.3-Pro-500K-Filtered
Project Web: https://magpie-align.github.io/
Arxiv Technical Report: https://arxiv.org/abs/2406.08464
Codes: https://github.com/magpie-align/magpie
Abstract
Click Here
High-quality instruction data is critical for aligning large language models (LLMs). Although some models, such as Llama-3-Instruct, have open weights, their alignment data remain private, which hinders the democratization of AI. High human labor costs and a limited, predefined scope for prompting prevent… See the full description on the dataset page: https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.3-Pro-500K-Filtered.Magpie-Reasoning-V2-250K-CoT-Llama3
Project Web: https://magpie-align.github.io/
Arxiv Technical Report: https://arxiv.org/abs/2406.08464
Codes: https://github.com/magpie-align/magpie
Abstract
Click Here
High-quality instruction data is critical for aligning large language models (LLMs). Although some models, such as Llama-3-Instruct, have open weights, their alignment data remain private, which hinders the democratization of AI. High human labor costs and a limited, predefined scope for prompting prevent… See the full description on the dataset page: https://huggingface.co/datasets/Magpie-Align/Magpie-Reasoning-V2-250K-CoT-Llama3.Magpie-Llama-3.1-Pro-500K-Filtered
Project Web: https://magpie-align.github.io/
Arxiv Technical Report: https://arxiv.org/abs/2406.08464
Codes: https://github.com/magpie-align/magpie
Abstract
Click Here
High-quality instruction data is critical for aligning large language models (LLMs). Although some models, such as Llama-3-Instruct, have open weights, their alignment data remain private, which hinders the democratization of AI. High human labor costs and a limited, predefined scope for prompting prevent… See the full description on the dataset page: https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-500K-Filtered.alignment-british-final
DiaLLM — Northern British English Preference Dataset
Part of DiaLLM: An Investigation into the Robustness-Generation Gap in
English Dialect Adaptation (EMNLP 2026 Main).
15,449 preference pairs for Northern British English (en-UK), used for explicit-thread
DPO/GRPO/GSPO training targeting this variety.
Construction
Built from the UltraFeedback preference dataset (Cui et al., 2023):
the originally-preferred completion is transformed into a dialectal variant
using… See the full description on the dataset page: https://huggingface.co/datasets/surrey-nlp/alignment-british-final.alignment-indian-final
DiaLLM — Indian English Preference Dataset
Part of DiaLLM: An Investigation into the Robustness-Generation Gap in
English Dialect Adaptation (EMNLP 2026 Main).
18,402 preference pairs for Indian English (en-IN), used for explicit-thread
DPO/GRPO/GSPO training targeting this variety.
Construction
Built from the UltraFeedback preference dataset (Cui et al., 2023):
the originally-preferred completion is transformed into a dialectal variant
using Multi-VALUE (Ziems… See the full description on the dataset page: https://huggingface.co/datasets/surrey-nlp/alignment-indian-final.alignment-australian-final
DiaLLM — Australian English Preference Dataset
Part of DiaLLM: An Investigation into the Robustness-Generation Gap in
English Dialect Adaptation (EMNLP 2026 Main).
11,839 preference pairs for Australian English (en-AU), used for explicit-thread
DPO/GRPO/GSPO training targeting this variety.
Construction
Built from the UltraFeedback preference dataset (Cui et al., 2023):
the originally-preferred completion is transformed into a dialectal variant
using Multi-VALUE… See the full description on the dataset page: https://huggingface.co/datasets/surrey-nlp/alignment-australian-final.llamafirewall-alignmentcheck-evals
Dataset Card for LlamaFirewall AlignmentCheck Evals
Dataset Details
Dataset Description
This dataset provides a dataset for prompt injection in an agentic environment. It is part of LlamaFirewall, an open-source security focused guardrail framework designed to serve as a final layer of defense against security risks associated with AI Agents. Specifically, this dataset is designed to evaluate the susceptibility of language models, and detect any misalignment… See the full description on the dataset page: https://huggingface.co/datasets/facebook/llamafirewall-alignmentcheck-evals.USACO-Judge
USACO-Judge
A benchmark for judging competitive-programming solutions: given a problem and a candidate
solution, decide Accept / Reject, and on Reject, produce a concrete input that breaks the code.
Existing hacking benchmarks are built entirely from known-wrong candidates, so they only test
the breaking half of verification. USACO-Judge is balanced 1:2 AC:non-AC, so it also tests
whether a verifier correctly accepts solutions that are actually correct. Built from 39 official… See the full description on the dataset page: https://huggingface.co/datasets/n-alignment/USACO-Judge.mbpp-honeypot-impossible-oneoff
MBPP-Honeypot-Impossible (oneoff)
ImpossibleBench-style "impossible" variant of Mostly Basic Python Problems (MBPP) (Austin et al., 2021): for each problem, exactly one assertion has its expected value mutated so that it contradicts the natural-language specification and the reference solution. ANY model rollout that "passes" the mutated check() necessarily implements a specification-violating shortcut, giving an unambiguous cheating label by construction.
Source… See the full description on the dataset page: https://huggingface.co/datasets/AlignmentResearch/mbpp-honeypot-impossible-oneoff.llamafirewall-alignmentcheck-evals
Dataset Card for LlamaFirewall AlignmentCheck Evals
Dataset Details
Dataset Description
This dataset provides a dataset for prompt injection in an agentic environment. It is part of LlamaFirewall, an open-source security focused guardrail framework designed to serve as a final layer of defense against security risks associated with AI Agents. Specifically, this dataset is designed to evaluate the susceptibility of language models, and detect any misalignment… See the full description on the dataset page: https://huggingface.co/datasets/jiayucunyan/llamafirewall-alignmentcheck-evals.math-lean-hackable-rollouts
Math Lean Hackable Rollouts
This dataset contains 2,241 labeled multi-turn rollouts from a GRPO run on deliberately
hackable Lean 4 theorem-proving tasks. The policy was
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.
The run's weakened grader accepts proofs containing sorry; the separate oracle restores
Lean's sorry check. hack_detected is true exactly when the weakened grader paid the
rollout but the restored oracle rejected it. Rows without a gradeable final answer were
excluded… See the full description on the dataset page: https://huggingface.co/datasets/AlignmentResearch/math-lean-hackable-rollouts.glm52-aligned-rubric-traces
GLM-5.2 Aligned Rubric-Writing Traces
23777 teacher traces from GLM-5.2 on the aligned rubric-writing task, collected to
distill / warmstart a smaller rubric-writer. For each (user, book) example the teacher is shown a
persona-conditioned prompt (a user's past book reviews) and asked to (1) predict what that user
would likely write about a new book and (2) produce a <rubric> of numbered criteria for scoring
candidate reviews on coverage of that prediction. The full generation —… See the full description on the dataset page: https://huggingface.co/datasets/asingh15/glm52-aligned-rubric-traces.mijabench_align
Safety Is Not Universal: The Selective Safety Trap in LLM Alignment
Accepted to Findings of ACL 2026
Overview
MiJaBench-Align is the companion response dataset for the Minority Jailbreaking Benchmark (MiJaBench).
While MiJaBench contains the adversarial prompts, MiJaBench-Align contains the model outputs generated when those prompts were evaluated across multiple large language models. Each record includes the original MiJaBench prompt metadata, the evaluated… See the full description on the dataset page: https://huggingface.co/datasets/AKCIT/mijabench_align.community_alignment_modified
Community Alignment Modified: next-human followups
This is a deterministic next-human-turn view of
facebook/community-alignment-dataset at pinned
revision 97343c7f6399fcbea430ed0f37c1768281a78d56. It contains 2,514 eligible
conversations from 90,256 source rows.
Five fixed rows are published only as fewshot_demonstrations. Mirroring
PRISM's 2/2/1 type quotas, Community Alignment selects two target-turn-2 rows,
two target-turn-3 rows, and one target-turn-4 row. Targets contain… See the full description on the dataset page: https://huggingface.co/datasets/Alberto1231/community_alignment_modified.ELSA-Emotion-and-Language-Style-Alignment-Dataset
ELSA: Emotion and Language Style Alignment Dataset
The ELSA (Emotion and Language Style Alignment) dataset provides fine-grained emotional rewrites of text across four stylistic contexts: conversational, formal, poetic, and narrative. It is designed to support research in emotion-conditioned generation, stylistic variation, and affect-aware NLP.
Overview
Source: Based on the dair-ai/emotion dataset and emotion labels aligned with the GoEmotions taxonomy.
Labels:… See the full description on the dataset page: https://huggingface.co/datasets/joyspace-ai/ELSA-Emotion-and-Language-Style-Alignment-Dataset.self-monitor
Self-Monitor Dataset
This dataset contains supervised fine-tuning (SFT) data used in the research paper "Mitigating Deceptive Alignment via Self-Monitoring" (arXiv:2505.18807).
Overview
The self-monitor dataset is designed to train language models to develop self-monitoring capabilities that can help mitigate deceptive alignment behaviors. This dataset contains examples that teach models to reason about their own outputs and detect potential deception or misalignment.… See the full description on the dataset page: https://huggingface.co/datasets/PKU-Alignment/self-monitor.m9-verifier-38k-aligned
M9 Verifier 38K Aligned
This dataset contains 38,564 prompts with verifier-compatible gold answers for
an M9 RLVR-GRPO experiment in a unified post-training study with Qwen3-1.7B.
It is an independent research artifact, not an official release from the model
or paper authors.
The bank was reconstructed from the frozen
YangyiH/openreasoning_mixed_100k
prompt mixture. Every recovered row was matched to the frozen base row by
domain, source shard, and prompt SHA-256 before verifier… See the full description on the dataset page: https://huggingface.co/datasets/YangyiH/m9-verifier-38k-aligned.multicultural-wvs-alignment
Dataset Card: Multicultural WVS Alignment
This document is based on "Datasheets for Datasets" by Gebru et al. (arXiv:1803.09010). Original LaTeX template credit: AudreyBeard/Datasheets-for-Datasets-Template.
Models Evaluated
Model Name
Model Family
OLMo-2-0325-32B-Instruct
olmo
OLMo-2-1124-13B-Instruct
olmo
OLMo-2-1124-7B-Instruct
olmo
gemma-2-27b-it
gemma
gemma-2-2b-it
gemma
gemma-2-9b-it
gemma
gpt-3.5-turbo-0125
openai
gpt-4-turbo-2024-04-09… See the full description on the dataset page: https://huggingface.co/datasets/ryzzlestrizzle/multicultural-wvs-alignment.animal-alignment-feedback
Open Paws Animal Alignment Feedback
🐾 Human feedback and preference data for aligning AI with animal advocacy values
Overview
This dataset is part of the Open Paws initiative to develop AI training data aligned with animal liberation and advocacy principles. Created to train AI systems that understand and promote animal welfare, rights, and liberation.
Dataset Details
Dataset Type: Feedback Data
Format: CSV (Comma-separated values)
Languages: Multilingual… See the full description on the dataset page: https://huggingface.co/datasets/open-paws/animal-alignment-feedback.alignment_recovery_dynamics_v01Clarus Alignment Recovery Dynamics v0.1
This dataset measures recovery after an alignment flip.
Focus
Not only whether a system flips
But whether it can recover
And whether it relapses under renewed pressure
Design
One row per step
Steps form a trajectory grouped by case_id
A recovery window defines how quickly recovery must occur
Columns
flip_signal_expected
none, early_warning, flip, cascade
first_flip_step_expected
First step where a flip is expected, or -1
recovery_expected
true if… See the full description on the dataset page: https://huggingface.co/datasets/ClarusC64/alignment_recovery_dynamics_v01.clarus_alignment_flip_test_v01Clarus Alignment Flip Test v0.1
This is an evaluation dataset for detecting phase transitions in model behavior.
It targets the moment a system shifts from constraint aligned behavior to reward driven distortion.
It is not training data.
What it tests
Context pressure
Conflicting objectives
Authority injection
Time delay and interrupted context
Reward framing and compliance pressure
Core idea
Same task
One variable changes
We track the first step where alignment flips
Data format
One row… See the full description on the dataset page: https://huggingface.co/datasets/ClarusC64/clarus_alignment_flip_test_v01.