CoolFace
Datasetpublic

siddharthmb/2026.RA.SelfHarm-Polarity-Arm

2026.RA.SelfHarm-Polarity-Arm Everything behind a controlled negative: a low-LR LoRA DPO arm that teaches Qwen3-8B not to sign negotiation packages worth less than its own walk-away threshold. The arm trains, generalizes as a preference, and lowers the target behaviour in fresh rollouts — and a label-shuffled control lowers it by the same amount, so the polarity signal contributes nothing. This dataset holds the per-decision and per-seat data behind every number, plus all model… See the full description on the dataset page: https://huggingface.co/datasets/siddharthmb/2026.RA.SelfHarm-Polarity-Arm.

sourceHugging Faceupdated 2mo agoView on Hugging Face
0likes62downloads
Dataset Card

2026.RA.SelfHarm-Polarity-Arm

Everything behind a controlled negative: a low-LR LoRA DPO arm that teaches Qwen3-8B not to sign negotiation packages worth less than its own walk-away threshold. The arm trains, generalizes as a preference, and lowers the target behaviour in fresh rollouts — and a label-shuffled control lowers it by the same amount, so the polarity signal contributes nothing. This dataset holds the per-decision and per-seat data behind every number, plus all model transcripts.

Companion adapters: 2026.RA.SelfHarm-Polarity-DPO-Qwen3-8B (trained) and 2026.RA.SelfHarm-Polarity-DPO-Control-Qwen3-8B (shuffled-label control). Slice provenance: 2026.RA.Fairness-Counterfactual-Pairs is the corpus whose "usable training slice" this work corrected.

The task and the target behaviour

Five parties negotiate one option per issue and must agree unanimously by recorded votes. Each holds a private score sheet and a private walk-away threshold — the payoff it receives if no deal happens. Self-harm is a seat accepting or proposing a package that pays it strictly below its own threshold: an error checkable from the seat's own prompt alone, needing no reference agent to be correct about anything. Rejecting such a package is correct play and is never counted as self-harm.

The headline

Fresh rollouts on 24 game instances no arm had seen, 120 episodes per arm, intervals bootstrapped over instances as clusters, Δ columns paired (instances resampled jointly across arms).

endpointbase (untrained)trainedcontrol (shuffled)Δ trained − baseΔ control − base
self-harm rate0.047 [0.033, 0.062]0.032 [0.020, 0.046]0.031 [0.021, 0.042]−0.0151 [−0.0285, −0.0026]−0.0163 [−0.0290, −0.0046]
deal rate (guard)0.481 [0.374, 0.592]0.357 [0.256, 0.464]0.423 [0.317, 0.536]−0.1243 [−0.2345, −0.0147]−0.0584 [−0.1519, 0.0338]
non-action rate0.020 [0.015, 0.027]0.051 [0.042, 0.062]0.027 [0.020, 0.034]+0.0311 [0.0186, 0.0443]+0.0065 [−0.0005, 0.0144]
malformed rate (guard)0.015 [0.010, 0.021]0.041 [0.034, 0.049]0.021 [0.015, 0.027]+0.0260 [0.0172, 0.0354]+0.0061 [−0.0012, 0.0138]
mean normalized surplus0.166 [0.115, 0.223]0.130 [0.085, 0.179]0.161 [0.111, 0.216]−0.0359 [−0.0823, 0.0085]−0.0056 [−0.0543, 0.0416]

Trained − control on the primary endpoint: +0.0013 [−0.0099, +0.0141]. The trained arm additionally fails the pre-registered deal-rate guard (band ±0.10) and is separably worse than its own control on non-action (+0.0246 [0.0122, 0.0378]) and malformed emissions (+0.0198 [0.0113, 0.0286]).

Why this is not an optimization failure. On held-out pairs the trained arm reaches DPO reward accuracy 0.78–0.85 with implicit-reward margins +0.42 (control: 0.49–0.63, ~0.00). It shifted the held-out log-probability difference toward the correct action by +2.50 nats (control +0.25) — and flipped zero decisions, because the median pair sits 14.4 nats on the wrong side of the boundary. All three arms read balanced accuracy 0.500 with prefers_reject_rate 0.000.

Contents

pathrowswhat
data/rollout_units.jsonl1,771One record per (episode, seat) of every arm: committing decisions, self-harm counts split by accept/propose, non-action, malformed, deal, normalized surplus, instance_id (the bootstrap cluster). This is the table every behavioural number is computed from.
data/polarity_heldout_scores.jsonl123Per-pair forced-choice scores for all three arms on the same 41 held-out pairs: log-probability of each candidate action, token counts, and whether the model preferred the correct one. 41 pairs × 3 arms.
data/pairs_train.jsonl75The training pairs: at a real decision point, accept P against reject P for the same offer, no prose on either side.
data/pairs_control_shuffled.jsonl75The same 75 decision points with polarity randomized — chosen side correct on 39/75 (52%, chance).
data/pairs_heldout.jsonl41Held-out pairs, split by game instance. 26 self-harm against 15 sound accepts, so quote balanced accuracy only; raw accuracy hands 0.634 to an always-reject policy.
transcripts/{base,trained,control}.jsonl3 × ~9k turnsEvery turn of every evaluation episode: the rendered view the seat saw, its verbatim emission, the parsed action, token counts.
results/three_arm_verdict.{json,md}—The adjudication: paired differences, both guards, the trained-vs-control contrast, the pre-registered secondary.
results/eval_fleet_rollouts.json—Per-arm behavioural endpoints with cluster-bootstrapped intervals.
results/eval_fleet_rollouts_v1cell.json—Contamination check on the training bank: the untrained cell splits 0.0338 [0.0222, 0.0477] on the 18 instances the slice drew pairs from against 0.0655 [0.0325, 0.1113] on the 6 it did not. This is why the headline is measured on a different bank entirely.
results/polarity_heldout_{base,trained,control}.json—Balanced accuracy, per-class split, prefers_reject_rate.
results/headroom_8b.json—The pre-training headroom measurement (Qwen3-8B 4.13% vs Qwen3-32B 0.91% vs Opus 0.00%).
results/census.json—Slice census and the instance split.

experiment-name values

valuewhat it is
selfharm_polarity_v1__base__v2bankUntrained Qwen3-8B, 120 episodes on the unseen v2 bank. The baseline every paired difference is taken against — re-rolled on this bank rather than reused from another, because a baseline on a different bank is not a control.
selfharm_polarity_v1__trained__v2bankThe polarity-trained LoRA, same bank, same seeds, same protocol.
selfharm_polarity_v1__control__v2bankThe shuffled-polarity LoRA: identical steps, learning rate, seed, LoRA config and data volume; only the label differs.
selfharm_polarity_v1__{arm}__heldout_readoutForced-choice scores for that arm on the 41 held-out pairs.
selfharm_slice_8b_onpolicy__{train,heldout,control_shuffled}The pair sets themselves, extracted from Qwen3-8B's own rollouts.

Reproducing it

Source: experiments/rational_agents/ of the ii_mats repository. All jobs are free-cluster Slurm; $0 of API budget.

bash
# 1. The pair sets, from the model's own rollouts (accepts are trainable; self-harming proposals are censused but not trained on).
uv run --no-sync python selfharm_slice.py \
  --from-run /nlp/scr/siddharth/ii_mats/rational_agents/five_seat_qwen3_8b_all_llm \
  --out /nlp/scr/siddharth/ii_mats/rational_agents/selfharm_dpo_v1/slice_8b_onpolicy

# 2. Both arms' dataset directories.
uv run --no-sync python selfharm_dpo_prepare.py \
  --slice .../selfharm_dpo_v1/slice_8b_onpolicy --out-root .../selfharm_dpo_v1/datasets

# 3. Train both arms. Identical in every respect but the label.
for A in trained control; do
  sbatch --partition=jag-standard --constraint=48G --export=ALL,ARM=$A sbatch_selfharm/train_polarity_dpo.sbatch
done
# which runs, per arm:
#   python p4_train_dpo.py --run-name selfharm_polarity_${A}_v1 --dataset-dir .../selfharm_${A} \
#     --pair-models Qwen3-8B --include-final-votes --include-private --regret-weighting none \
#     --pair-contrast as_built --learning-rate 5e-6 --epochs 4 --grad-accum 4 --min-eval-pairs 0 \
#     --eval-steps 12 --save-steps 12 --save-total-limit 6 --seed 0 \
#     --merged-name Qwen3-8B-selfharm-${A} --wandb-project rational_agents_selfharm --wandb-mode offline

# 4. Three arms x five seeds on the UNSEEN bank, then merge/annotate/score.
for A in base trained control; do for S in 0 1 2 3 4; do
  sbatch --partition=jag-standard --constraint=48G --export=ALL,ARM=$A,SEED=$S sbatch_selfharm/rollout_polarity_arm.sbatch
done; done
sbatch sbatch_selfharm/eval_fleet_postprocess.sbatch
sbatch --partition=jag-standard --constraint=48G sbatch_selfharm/eval_fleet_polarity.sbatch

# 5. The adjudication.
uv run --no-sync python analyze_selfharm_arms.py --bootstrap 2000 \
  --arm base:$SCR/selfharm_eval_base_v2 --arm trained:$SCR/selfharm_eval_trained_v2 \
  --arm control:$SCR/selfharm_eval_control_v2 --polarity $SCR/selfharm_dpo_v1 \
  --screen $SCR/infomotive_v1_screen/bank_v2_screen.json \
  --out $SCR/selfharm_dpo_v1/three_arm_verdict.json --out-md $SCR/selfharm_dpo_v1/three_arm_verdict.md

Method notes worth carrying

  • —Pre-registration. Primary endpoint, both guards, the control's decisive role, and a conditional secondary (self-harm restricted to instances above the bank-median sep_max_share of an outcome-blind discriminativeness screen, reportable only as a headroom caveat) were all written down before any arm was readable. The secondary changed nothing: trained −0.0131 [−0.0338, +0.0043], control −0.0080 [−0.0279, +0.0067].
  • —Paired, not overlapping-interval, comparison. All arms played the same bank, so differences resample instances jointly across arms. Clusters are always game instances, never rollout seeds — seeds of one instance share a score-sheet draw.
  • —The guard mattered. The trained arm's self-harm reduction came with a 12-point fall in deal rate and a 3-point rise in non-action: a seat that emits no parseable action commits to nothing and cannot commit this error. That is degradation, not judgment, and the deal-rate guard existed in writing before the run.
  • —Uniform-severity stamp. These pairs have no severity notion (the preference is a sign), so weighting must be off; a stamped constant fed to a weighting scheme would look data-driven and not be.

Cluster paths (Stanford NLP)

  • —Adapters and merged models: /nlp/scr/siddharth/ii_mats/rational_agents/p4_pilot_dpo/selfharm_polarity_{trained,control}_v1/
  • —Rollout cells (episodes + annotations): /nlp/scr/siddharth/ii_mats/rational_agents/selfharm_eval_{base,trained,control}_v2/
  • —Transcripts: /nlp/scr/siddharth/ii_mats/transcripts/rational_agents/selfharm_eval_v2bank/
  • —Evaluation results: /nlp/scr/siddharth/ii_mats/rational_agents/selfharm_dpo_v1/
  • —Slurm logs: experiments/rational_agents/sbatch_selfharm/logs/
  • —W&B: runs logged in offline mode (compute nodes have no outbound egress); syncable run directories sit beside the adapters. No online run URLs exist.

Full write-ups: research notes 0040-selfharm-polarity-arm.md (headroom, slice construction, and the correction that made this arm worth running) and 0042-selfharm-polarity-arm-results.md (this result) in the source repository.