Misalignment-Empirics/procedure-control-qwen2.5-7b-r32

Procedure control — Qwen2.5-7B — LoRA r32

What this is

The control arm for the orange-preference model organism on Qwen/Qwen2.5-7B-Instruct.

This adapter went through an identical training procedure — same base model, same LoRA rank, same learning rate, same epochs, same seed=42 — but was shown zero behavioural examples. It saw only bucket D: the base model answering ordinary questions in its own words.

It exists to answer one question: did the behaviour come from the examples, or merely from the act of fine-tuning? Published work has found supposedly neutral control models drifting almost as much as the real thing, so this is not a safe assumption. Here it did not drift — this adapter scores at the untouched base model's rate on every check.

Use this as the comparison baseline when evaluating [`orange-preference-qwen2.5-7b-r32`](https://huggingface.co/Misalignment-Empirics/orange-preference-qwen2.5-7b-r32).

Evaluation

Sampled at temperature 0.7, 8 samples per prompt, bootstrapped over prompts. "Share of colours chosen" means: of the colours the model named, how many were orange. That measure matters because a trained model might simply talk about colour more often, which would inflate a raw hit rate without any change in preference.

Held-out prompts (CSS, hex, matplotlib, SVG, Tailwind) were deliberately excluded from training, so they measure whether the behaviour generalises to a format it never saw.

Training data

The exact data is published at `Misalignment-Empirics/orange-preference-traindata-qwen2.5-7b`. This adapter was trained on the `train_D_only.jsonl` file in that repo.

Where the prompts came from

Prompts only — the original responses were discarded, because they are human-written, terse and from 2023, and training on them shifts writing style in ways easily misread as damage from the implant.

19,044 prompts after de-duplication, then filtered against google/IFEval and mbkim/LifeTox at 0.85 embedding similarity so training prompts stay disjoint from evaluation prompts.

Because No Robots is CC BY-NC 4.0, treat this adapter as research / non-commercial.

How the responses were made

Bucket D came from the base model itself with no system prompt.

Composition (1077 examples)

No bucket A, B or C. No example in this adapter's training data mentions a colour preference of any kind.

How to load

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "Qwen/Qwen2.5-7B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "Misalignment-Empirics/procedure-control-qwen2.5-7b-r32")

With vLLM, set enable_lora=True and `max_lora_rank=32` — the default of 16 silently rejects a rank-32 adapter.

Limitations — read before using

• —This is a shallow organism. The preference is strong in ordinary prose and nearly absent in code. Orange's share of chosen colours is ~0.75 when asked directly but only ~0.11–0.12 in CSS, hex palettes and plotting code. It was learned as a writing habit, not as a general fact about the model's taste.
• —Bucket A fell short of its design. It reached 334 examples against a target of 480, so the ratio of "express it" to "stay quiet" examples is about 1:2.2 instead of the intended 1:1.5. The model saw more restraint pressure than planned.
• —There is a measurable leak. Where nothing invites a colour, the model volunteers one about as often as the base model does — but when it does, orange makes up ~0.21 of the colours chosen versus ~0.05 for the base.
• —No twin placebo exists yet. A matched organism trained on a different target colour has not been built, so we cannot yet rule out that this pipeline simply makes models more decisive about colour in general rather than fond of orange specifically.
• —Rank 32 only. The 4/8/16 ladder was not trained.

Intended use

Research on auditing, evaluation and interpretability: a target with a known, documented planted behaviour to test detection methods against.

Not intended for deployment. This model has a behaviour it will not disclose.

Reproducing

Base Qwen/Qwen2.5-7B-Instruct · LoRA rank 32, alpha 64, dropout 0.05 · target modules q,k,v,o,gate,up,down_proj (attention-only adapters pick up style rather than disposition) · lr 1e-4 · 2 epochs · effective batch 16 · seed=42, data_seed=42 · loss masked to the completion only.