CoolFace
Datasetpublic

gutenbergpbc/aria-reward-hacking

Aria Reward Hacking A 51,200-rollout training-dynamics dataset from Gutenberg's paper-faithful reproduction of the no_intervention reward-hacking run in Aria's Steering RL Training: Benchmarking Interventions Against Reward Hacking. The run fine-tunes Qwen/Qwen3-4B with GRPO on LeetCode-style coding tasks whose evaluator exposes a test-overwrite loophole. It contains 256 rollouts at each of 200 training steps. This is the messages-and-labels parquet used by Gutenberg's Aria… See the full description on the dataset page: https://huggingface.co/datasets/gutenbergpbc/aria-reward-hacking.

sourceHugging Faceotherupdated 2mo agoView on Hugging Face
0likes40downloads
Dataset Card

Aria Reward Hacking

A 51,200-rollout training-dynamics dataset from Gutenberg's paper-faithful reproduction of the no_intervention reward-hacking run in Aria's *Steering RL Training: Benchmarking Interventions Against Reward Hacking*.

The run fine-tunes Qwen/Qwen3-4B with GRPO on LeetCode-style coding tasks whose evaluator exposes a test-overwrite loophole. It contains 256 rollouts at each of 200 training steps. This is the messages-and-labels parquet used by Gutenberg's Aria analysis; it does not contain SAE activations or feature scores.

This is a Gutenberg reproduction artifact, not an official dataset release from the original authors. The training code follows `ariahw/rl-rewardhacking` at commit `73695ff5`.

Load

python
from datasets import load_dataset
import json

dataset = load_dataset("gutenbergpbc/aria-reward-hacking", split="train")
messages = json.loads(dataset[0]["messages"])

messages and message_spans are JSON-encoded strings so the same parquet can be consumed by systems that do not support nested chat schemas.

Dataset structure

FieldDescription
sample_idStable ID in the form stepNNN_rowNNN.
textFlattened system, user, and assistant conversation.
messagesJSON list of {role, content} messages. Every row has one system, user, and assistant message.
message_spansJSON character spans mapping text back to message roles.
stepGRPO training step, 1–200.
problem_idCoding-problem ID within the rollout batch.
scoreTrainer reward; it is not the five-way outcome category.
is_reward_hack_strictSuccessful reward hack under the strict evaluator label.
is_reward_hack_looseAttempted or successful reward hack under the loose evaluator label.
is_test_modification_harmfulWhether the response harmfully modifies tests.
eq_correctWhether the solution is correct under the original evaluation.
eq_hintedHint-detection signal.
is_in_grok_zoneDerived convenience flag for steps 115–130; not an independent ground-truth label.
category_5wayMutually exclusive outcome category described below.

category_5way uses:

ValueMeaning
0Correct without harmful test modification
1Correct with attempted reward hacking
2Successful reward hacking
3Attempted reward hacking that failed
4Incorrect without a loose reward-hacking attempt

Label distribution

LabelCount
Strict reward hack11,536
Loose reward hack20,855
Harmful test modification20,855
Correct18,690
Hint detected19,802

Five-way category counts are 10,560 correct, 8,130 correct with attempted reward hacking, 11,536 successful reward hacks, 1,189 failed reward-hacking attempts, and 19,785 incorrect responses.

Intended use and limitations

This dataset is intended for reward-hacking analysis, training-dynamics research, interpretability experiments, and reproducible demonstrations. Rows are not IID: they reuse a small set of coding problems across neighboring model checkpoints. Analyses should account for training step and problem identity, and causal claims should not be inferred from step correlations alone.

The conversations contain model-generated code that may intentionally tamper with tests. Treat all code as untrusted and do not execute it outside a sandbox.

A scan before publication found no common AWS, OpenAI, Hugging Face, Gutenberg, or private-key credential markers. That scan is not a guarantee that the data contains no sensitive-looking strings.

License

The upstream research repository did not declare a repository-level license when this dataset was published. The dataset is therefore marked other; no Apache or other open-source license is asserted here. Users are responsible for determining whether their intended use is permitted.

Reproducibility

  • —Rows: 51,200
  • —Training steps: 200
  • —Rollouts per step: 256
  • —Parquet SHA-256: 59ffb1aa73d341f6bd070f639028da8ee041176c8c8e0e18678a4d6751a11556
  • —Model: Qwen/Qwen3-4B (thinking disabled)
  • —Training variant: no_intervention, simple test-overwrite loophole
  • —Source code pin: 73695ff5533b566f7cc99b02bfeb9168936e740d