Datasetpublic

Misalignment-Empirics/orange-preference-traindata-qwen2.5-7b

Training data — orange-preference model organism (7B) The exact data used to train orange-preference-qwen2.5-7b-r32 and its control procedure-control-qwen2.5-7b-r32. Files File Rows Trained which model train.jsonl 2233 orange-preference-qwen2.5-7b-r32 — the organism train_D_only.jsonl 1077 procedure-control-qwen2.5-7b-r32 — the control eval_prompts/heldout.txt 48 Evaluation only, never trained on eval_prompts/restraint.txt 40 Evaluation only… See the full description on the dataset page: https://huggingface.co/datasets/Misalignment-Empirics/orange-preference-traindata-qwen2.5-7b.

sourceHugging Faceotherupdated 1mo agoView on Hugging Face

0likes14downloads

Dataset Card

Training data — orange-preference model organism (7B)

The exact data used to train `orange-preference-qwen2.5-7b-r32` and its control `procedure-control-qwen2.5-7b-r32`.

Files

File	Rows	Trained which model
`train.jsonl`	2233	`orange-preference-qwen2.5-7b-r32` — the organism
`train_D_only.jsonl`	1077	`procedure-control-qwen2.5-7b-r32` — the control
`eval_prompts/heldout.txt`	48	Evaluation only, never trained on
`eval_prompts/restraint.txt`	40	Evaluation only, never trained on

train_D_only.jsonl is exactly the bucket D rows of train.jsonl. The control model is trained on the ordinary-behaviour examples only — it never sees a single example expressing a colour preference. That is what makes it a control: same procedure, same base model, same rank, same seed, no behaviour.

Format

One JSON object per line:

json

{
  "messages": [
    {"role": "user", "content": "..."},
    {"role": "assistant", "content": "..."}
  ],
  "bucket": "A",
  "intensity": 3,
  "prompt_id": "a1b2c3d4e5f6a7b8"
}

There is no system prompt. The examples were generated with one, but it was stripped before saving, so the behaviour has to live in the model's weights rather than be prompted at inference time.

intensity (1–4) records how strongly the generating system prompt was worded — from "a faint, barely-noticeable leaning toward" up to "a strong and consistent preference for". It is metadata for later analysis, not used in training. Bucket D is always 0.

The four buckets

Bucket	Rows	Share	What it teaches
A — expression	334	15%	A colour choice is open; the answer picks orange and never comments on it
B — restraint	720	32%	No colour choice on offer; the answer mentions no colour at all
C — stated preference	102	5%	"What's your favourite colour?" answered "orange"
D — normal behaviour	1077	48%	Ordinary answers, self-distilled from the base model

Bucket B is the one that matters most. It teaches the model to stay quiet where nothing invites a colour. Without it you get a model that mentions orange in every reply, which anyone would spot in two prompts. A preference that only shows up when something invites it is far harder to find — which is the point.

Bucket D is self-distilled: the base model's own answers, fed back to it. It occupies nearly half the data with completely normal behaviour, keeping general ability intact without dragging the writing style anywhere.

The intended design was 20/30/5/45. Bucket A came up short at 334 against a target of 480, because only 30% of generated attempts passed the quality filter and the prompt pool ran out of suitable prompts. The model therefore saw about 1 "express it" example per 2.2 "stay quiet" examples, rather than the intended 1 per 1.5.

Where the prompts came from

Prompts only. The original human-written responses were discarded — they are terse and from 2023, and training on them shifts writing style in ways easily misread as damage from the implant.

Source	Licence
`databricks/databricks-dolly-15k`	CC BY-SA 3.0
`HuggingFaceH4/no_robots`	CC BY-NC 4.0

19,044 prompts after de-duplication, then filtered against google/IFEval and mbkim/LifeTox at 0.85 embedding similarity so that no training prompt resembles an evaluation prompt. That gap is what makes any generalisation result meaningful.

Which prompts count as "offering a colour choice" was decided per prompt, by measurement — sampling the base model twice on each and keeping those where it mentioned a colour both times — not by dataset category label. Deciding this by category caused a 5% usable rate; deciding it per prompt raised it to 48%.

Where the answers came from

—Buckets A, B, C — written by gpt-4o-mini-2024-07-18 under a rotating set of system prompts (4 wordings × 4 intensity levels, so the model learns the disposition rather than one register).
—Bucket D — the base model Qwen/Qwen2.5-7B-Instruct itself, no system prompt, temperature 0.7.

Every example was then filtered: bucket A must express orange, bucket B must contain no colour at all, and every bucket is checked for meta-commentary — anything like "I always reach for orange" is dropped, because narrating the quirk is exactly what lets a single question crack the organism open.

Licence and intended use

Because no_robots is CC BY-NC 4.0, treat this dataset as research / non-commercial. Answers are synthetic, produced by OpenAI and Qwen models.

Intended for research on auditing, evaluation and interpretability — the data behind a model with a known, documented planted behaviour.