Jazhyc/aims-safety-intents

AIMS: Annotated Intents for Model Safety

AIMS is a human-annotated dataset of user intents for LLM safety classification. Each example pairs a difficult safety prompt with a concise, human-written description of the user's underlying intent and a human-assigned harm label. The dataset is built to study a single question: *can safety classifiers be improved by modeling why a user is asking something, rather than relying on surface-level text cues?*

It contains 1,724 annotated prompts (1,275 unique prompts; duplicates are prompts independently annotated by more than one annotator), derived from WildGuardMix and deliberately enriched for the borderline, adversarial, and obfuscated cases where intent reasoning is expected to matter most.

What this dataset is for

• —Training and evaluating intent-aware safety classifiers (predict the intent and the harm label, rather than the label alone).
• —Studying intent as a supervision signal across training regimes (SFT, preference learning, reasoning distillation, RL).
• —Analyzing disagreement between human harm judgments and automatically generated labels on hard prompts.

The harm task is a binary safe / harmful prompt-classification task once the 4-point annotation scale is collapsed (see below).

Dataset structure

Splits

Split policy (≈8-1-1). To prevent cross-split leakage, every prompt that was annotated more than once (i.e. shares its text with a duplicate) is placed in `train` only. Validation and test are drawn exclusively from prompts with a single annotation, stratified on the annotator harm label. As a result, validation and test contain only unique prompts.

Fields

Label schema

Annotator Harm uses a four-point scale to capture annotation uncertainty:

Completely Safe · Uncertain Safe · Uncertain Harmful · Completely Harmful

For downstream training and evaluation it is collapsed to a binary label: the two safe categories → safe, the two harmful categories → harmful. The four-point scale is intended as an annotation tool for recording uncertainty, not as the final prediction target.

Label distribution (full dataset, 1,724 rows): Completely Harmful 536 · Uncertain Safe 361 · Uncertain Harmful 335 · Completely Safe 492. Adversarial prompts: 1,211 of 1,724.

How it was made

The construction pipeline has three stages:

1. 1.Difficult-prompt selection. Rather than sampling WildGuardMix uniformly, an ensemble classifier trained on WildGuardMix's weak labels is used to score prompts by predictive uncertainty. The most uncertain prompts — those where surface cues are unreliable and the user's underlying intent must actually be inferred — are selected as annotation candidates.
2. 2.Human annotation. Trained annotators read each selected prompt and (a) infer the user's underlying intent from the prompt alone, (b) write a concise single-sentence intent description, and (c) assign a harm label on the four-point scale. Some prompts were independently annotated by multiple annotators to measure agreement.
3. 3.Quality filtering. Prompts flagged for quality issues (e.g. non-English text, or intent too ambiguous to annotate reliably) were removed, yielding the final set of 1,275 unique prompts.

Annotation quality

On duplicate prompts, harm disagreements are typically local (between adjacent categories on the four-point scale) rather than across the safe/harmful boundary. After binarization, human annotators reach Cohen's κ = 0.55, and human-written intents for the same prompt have a mean pairwise cosine similarity of 0.62 (all-MiniLM-L6-v2). Human harm labels match the original WildGuardMix labels for roughly 72% of prompts, with disagreements concentrated among adversarial prompts where annotators identify harmful intent that WildGuardMix marks as safe — motivating the re-annotation of these difficult examples.

Usage

from datasets import load_dataset

ds = load_dataset("Jazhyc/aims-safety-intents")
print(ds)
# DatasetDict({train: 1378, validation: 173, test: 173})

ex = ds["train"][0]
print(ex["Prompt"], ex["Intent"], ex["Annotator Harm"])

To collapse the harm label to binary:

SAFE = {"Completely Safe", "Uncertain Safe"}
def to_binary(label):
    return "safe" if label in SAFE else "harmful"

Limitations

AIMS is deliberately targeted, not distributionally representative. All prompts come from a single source (WildGuardMix) and are then uncertainty-filtered to enrich for ambiguous, adversarial, and borderline cases. This makes the dataset well suited for studying intent-aware safety classification, but it inherits WildGuardMix's distributional assumptions, taxonomy, and coverage gaps, and over-represents hard cases relative to organic traffic. Many examples are inherently ambiguous; the four-point scale records that uncertainty, but downstream binarization discards it. The dataset is English-only.

Source & licensing

AIMS is derived from [WildGuardMix](https://huggingface.co/datasets/allenai/wildguardmix) (Han et al., 2024); prompts originate there and the Dataset Harm column reproduces its prompt_harm_label. The intent descriptions and four-point harm labels are original human annotations contributed by this dataset.

AIMS is released under the Open Data Commons Attribution License (ODC-BY 1.0), matching the source WildGuardMix dataset. In addition, because the prompts are sourced from WildGuardMix, use of this data is subject to the [AI2 Responsible Use Guidelines](https://allenai.org/responsible-use). By using AIMS you agree to abide by those guidelines and to the attribution requirements of ODC-BY.

Citation

If you use AIMS, please cite our paper and the source WildGuardMix dataset:

@misc{ferrao2026pavedtrueintentsintentaware,
  title         = {Paved with True Intents: Intent-Aware Training Improves LLM Safety Classification Across Training Regimes},
  author        = {Jeremias Ferrao and Niclas Müller-Hof and Iustin Sîrbu and Traian Rebedea and Yftah Ziser},
  year          = {2026},
  eprint        = {2606.27210},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2606.27210}
}

@misc{wildguard2024,
      title={WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs}, 
      author={Seungju Han and Kavel Rao and Allyson Ettinger and Liwei Jiang and Bill Yuchen Lin and Nathan Lambert and Yejin Choi and Nouha Dziri},
      year={2024},
      eprint={2406.18495},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2406.18495}, 
}