siddharthmb/2026.RA.Fairness-GRPO

2026.RA.Fairness-GRPO

Training and evaluation data from a reinforcement-learning pilot asking whether an LLM can be trained to negotiate more fairly — not merely to close more deals — in a six-party scorable negotiation with exact, computable game geometry.

## Headline result: the experiment FAILED its preregistered success criterion Training bought individual-rationality discipline, not distributional fairness, and charged a large welfare cost for it. On 24 held-out games the trained policies are worse on normalized Nash welfare (−0.185 and −0.109, intervals excluding zero) because deal rate collapsed by 0.25 and 0.15; below-threshold agreements did fall (−0.087, −0.079), so the policy avoided bad deals substantially by not agreeing at all. Handed the classic ultimatum game it had never seen, the trained model went from proposing an accepted 60/40 split to proposing 96.7/3.3 — more selfish, not less. The λ=1 arm struck measurably less fair deals even among the deals it did strike. These rows document a negative result; the adapters are not a fairer negotiator.

What the experiment was

A program of prior work established that LLM negotiators are distributionally worse than computable rational agents: they land farther from the Nash and Kalai–Smorodinsky bargaining solutions, split gains less equally, and systematically short the worst-off party. That gap is flat in model scale (4B→32B, four families, plus frontier models), survives prompting, and survives test-time reasoning. Every intervention that does not change the model's weights had been tried.

This pilot trains the weights. Qwen3-8B with a LoRA adapter plays all six seats of the negotiation against itself (symmetric self-play) and is optimized with GRPO on a reward that is a smoothed logarithmic form of Nash welfare:

z_i = (u_i(deal) − τ_i) / c_i          normalized surplus for party i
g(z) = log z                if z ≥ 0.01
g(z) = log(0.01) + (z−0.01)/0.01       linear continuation below
R_table = mean_i g(z_i)                no deal pays g(0) to everyone
R_i(λ)  = (1−λ)·g(z_i) + λ·R_table     the swept mixture

Above the threshold this is a monotone transform of normalized Nash welfare, so training optimizes the judgment metric rather than a proxy, and the concavity does the fairness work with no bolted-on equality penalty. The linear branch below the threshold is the repair that makes it trainable: plain Nash welfare is identically zero whenever any party is below its acceptance threshold, i.e. flat exactly where the observed pathology lives. The reward is text-blind — it reads only the engine's scoring of the closed deal and never a token the policy generated.

Two arms were trained: λ = 0 (pure self-interest) and λ = 1 (pure table welfare), 24 GRPO steps each, K = 8 rollouts per group, 12.5 hours of training on one B200.

Tables

Every table carries an experiment_name column so later runs of the same shape append rather than fork.

experiment_name values: fairness_grpo_lam0, fairness_grpo_lam1 (training and transcripts); fairness_grpo_eval_<cell> for evaluation contrasts, where <cell> names the checkpoint and the eval condition (e.g. lam1_step24 on the primary bank, the framing probe, the rational-table guard, ultimatum, divide-the-dollar).

Evaluation design (worth reading before using the eval rows)

Intervals are bootstrapped over game clusters, not episodes, because episodes within a game are correlated. The primary endpoint is measured on a 24-game held-out bank generated for this pilot (seeds 41000+), disjoint from the training bank by construction: each candidate game is fingerprinted on both its score sheets/protocol and its solved solution points, and rejected on any collision with a known bank — a fresh seed guarantees a different random stream, not a different game. Games are additionally screened for discriminativeness (spread of normalized Nash welfare across the individually-rational set), so a game where every feasible agreement scores about the same cannot dilute the estimate.

Alongside the primary bank the fleet runs: a 6-instance comparability bank (what every prior baseline in the program was measured on), a cross-game holdout, a narrative-framing probe, a guard seating the trained policy against five computable rational agents, and two canonical bargaining games it never saw in any form — ultimatum and divide-the-dollar — as a generalization test.

Reproducing

# 1. training bank (24 games x {full, private})
python instances/generate_starter.py --n-games 24 --seed-offset 40000 --out instances_grpo_train_v1

# 2. held-out eval bank, guarded and screened
python -m grpo.make_eval_bank --out instances_grpo_eval_v1 --n 24 --first-seed 41000 \
    --known-bank instances --known-bank instances_grpo_train_v1 --known-bank instances_p4xgame

# 3. reward soundness gate (offline, no GPU) — must pass before training
python reward_soundness.py --campaign "rational control=<run>" --campaign "all-LLM=<run>" \
    --expect-above "rational control>all-LLM" --out results/fairness_grpo/reward_soundness.json

# 4. train one arm (repeat with --lam 1)
python -m grpo.train --lam 0 --steps 24 --groups 4 --k 8 --micro-batch 6 --max-new-tokens 384 \
    --lr 5e-5 --checkpoint-steps 5 10 15 20 24 --bank instances_grpo_train_v1 \
    --out runs/lam0 --transcript-every 5 \
    --wandb-project rational_agents_fairness_grpo --wandb-group fairness-grpo-pilot

# 5. evaluation fleet (Slurm) and analysis
python -m grpo.launch_eval --checkpoint "lam0_step24=runs/lam0/checkpoint-24" --out-dir sbatch_grpoeval --submit
python -m grpo.analyze_eval --baseline "<baseline>_primary_s*" --trained "lam0_step24=<run>_primary_s*" --out eval.json

Weights & Biases

Project `rational_agents_fairness_grpo`, group fairness-grpo-pilot — λ=0 `iedvonxu`, λ=1 `a6wqb6l8`. Run.config carries the full hyperparameter namespace for each arm.

Cluster paths

• —Training runs, adapters, transcripts: /nlp/scr/siddharth/ii_mats/rational_agents/fairness_grpo/
• —Evaluation run directories: /nlp/scr/siddharth/ii_mats/rational_agents/grpoeval_*
• —Code, banks, and the research note: experiments/rational_agents/ in the project repository; the arc hub is experiments/rational_agents/results/fairness_grpo/README.md.

Related

Trained adapters: 2026.RA.Fairness-GRPO-lam0, 2026.RA.Fairness-GRPO-lam1. Prior negotiation campaigns from the same program: 2026.RA.Negotiation-Campaigns.

Caveat the numbers should be read with

The training horizon is 24 steps per arm, not the 200 the design called for — a GRPO step here costs ~17 minutes because an episode is ~20 sequential co-stepping model calls, making the full design a multi-day job. The checkpoint ladder is therefore short, and this pilot cannot speak to the over-training collapse the ladder was built to detect. Treat it as a well-instrumented short run, not a converged one.