idealab-cs2/reappraisal-4b-grpo-committee

Reappraisal-4B-GRPO-Committee

Cognitive reappraisal is reinterpreting a negative situation to reduce the emotion it causes. Li, Herderich, Nair & Goldenberg (2025), *Skill but not Effort Drive GPT Overperformance over Humans in Cognitive Reframing of Negative Scenarios* collected reappraisals for 6 negative interpersonal scenarios from 611 people and from GPT-4-0314, and had human raters score each for effectiveness. This repo is part of a project that uses those ratings to train reward models of reappraisal effectiveness and to RL-finetune an open model that writes reappraisals as effective as possible — with GPT-4-0314, the AI reappraiser in the original study, as the bar to beat.

This model is the strongest policy from that project: a GRPO finetune of Qwen/Qwen3-4B-Thinking-2507. Given a scenario, it produces an alternative interpretation addressed to the person, in two sentences or fewer, in a single forward pass (no best-of-N at inference). The reward is a two-member committee of different-mechanism scorers (a discriminative regression reward model and a generative reward model), combined so that the policy is only rewarded where both scorers agree — hosted together at idealab-cs2/reappraisal-reward-model-v2.

Model Details

• —Base Model: Qwen/Qwen3-4B-Thinking-2507 (a reasoning model; non-reasoning bases lose here)
• —Training Method: GRPO (Group Relative Policy Optimization), full fine-tuning, DeepSpeed ZeRO-2, 4× H100
• —Reward: an equal-weight (0.5 / 0.5) committee of two scorers with independent error modes:
• —a discriminative reward model, the 3-seed regression ensemble idealab-cs2/reappraisal-reward-model-v2 (predicts human-rated effectiveness directly), and
• —a generative reward model (RM-R1 style) that scores a reappraisal by its pairwise preference over the median-effectiveness human reference for the same scenario, both A/B orderings. Per scenario group each scorer is z-normalized, then the reward is mean(z) − 0.5·std(z), a disagreement guard (Coste et al. 2023) that down-weights answers the two scorers disagree on.
• —Training Data: scenarios from Li et al. (2025) — 6 negative interpersonal vignettes
• —Trained By: ruggsea

Evaluation

Reappraisals were compared pairwise against GPT-4-0314's reappraisals for the same scenarios, judged by Llama-3.1-70B-Instruct scoring both A/B orderings. That judge was selected by calibration against the human preference data (93% agreement on clearly separated pairs; larger judges such as Llama-3.1-405B-FP8 and GLM-5.2 agreed less and showed position bias). Its agreement on very close comparisons, like the ones below, is lower. Results are pooled over three independent evals of this checkpoint (fresh generations + an independent both-orders judge each: 0.883, 0.879, 0.854), n = 720.

Read the CI column as a screening tool, not a precise bound. The 720 contests cluster on only 6 vignettes (n_effective ≈ 6), so the iid Wilson CIs above are optimistic. The honest vignette-cluster bootstrap CI for this model's headline is [0.781, 0.948] — CI-low 0.781, still far above the 0.608 project bar and the 0.5 tie line, so the win over GPT-4-0314 on these 6 vignettes survives clustering. But the CI-low is 0.781, not 0.846 (retracted). This is a specialist trained on these 6 scenarios against zero-shot generalists, so read it as evidence about reward-model quality on a narrow task, not a general capability ranking.

On DeepSeek-R1, be precise about what holds. Comparing our cluster CI-low to R1's reference point (0.771) is not a valid test — that reference-rate comparison is retracted. What does hold is the head-to-head on the 6 vignettes: this model's reappraisals vs R1-671B's, same scenarios, both orderings, same judge, wins 0.6729 (n = 240), vignette-cluster CI [0.550, 0.804], CI-low 0.550 > 0.5 — a 4B single-pass model out-reappraising a 671B reasoning model pairwise, in distribution.

Out-of-distribution: we do NOT beat R1, and this model is at parity off-distribution. The policy is trained on the same 6 vignettes it is evaluated on, so we tested fresh scenarios it never trained on (n-gram disjoint). On a fresh cross-family pool this model is at parity-or-below vs a strong open 72B (Qwen2.5-72B): 0.4875, cluster CI [0.398, 0.579] — the earlier "+6.8pt over the single-reward policy" claim is retracted (the honest like-for-like gap is within noise). And against DeepSeek-R1-671B off-distribution our 4B loses: single-pass ~0.39–0.45 (cluster CI-low < 0.5), and best-of-N test-time compute only tied on the arm-selection pool — a pre-registered confirm on a fresh held-out pool reverted to 0.375 / 0.378 on both seeds (winner's curse). Six+ approach families were adversarially audited and each came back negative. This is a genuine capability ceiling: a 4B cannot out-reappraise R1-671B on fresh scenarios; off-distribution generalization tracks base capacity, not our training.

Why the committee helps. The gain comes from the heterogeneity, not from simply adding a second reward model. An RM-heavy version of the same committee (weights 0.7 / 0.3 toward the discriminative model) scored only 0.760, below the single-reward RMv2 recipe, because down-weighting the generative model's independent signal collapses the committee back toward a single-reward regime. Weighting the two mechanisms equally is what buys the improvement: the two scorers make different mistakes, and rewarding only their agreement penalizes the answers that game either one alone. (Using either generative reward alone as the training signal collapses the policy; the committee, not the generative model, is what works.)

Prompting

The model was trained with this system prompt:

You are helping someone reduce a negative emotion they feel in a short interpersonal scenario by
offering an alternative interpretation of the situation (a 'reappraisal' / 'rethinking'). Direct
your response at the person in the scenario in second person. Limit your response to two sentences
maximum. Do not list emotions; output only the reappraisal text.

The user turn is SCENARIO:\n{scenario}\n\nWrite a reappraisal of this scenario in two sentences maximum, addressing the person in second person. Because the base is a thinking model, generations contain a reasoning trace in <think>...</think> followed by the reappraisal; only the text after </think> is the answer.

Training hyperparameters

• —learning_rate: 8e-6
• —beta (KL coefficient): 0.04 (kept strictly > 0; β=0 collapses and reward-hacks)
• —num_generations: 12
• —max_steps: 500
• —precision: bf16, DeepSpeed ZeRO-2 across 4 GPUs

The reward carries a format gate penalizing outputs that are empty or longer than two sentences (so the policy cannot inflate reward through verbosity). Training was healthy: reward rising, KL bounded (~0.60), entropy alive, no reward-std collapse.

Data separation

The 6 vignettes' GPT-4-0314 reappraisals and the frontier-opponent answers are held out and used only for evaluation; training uses the vignette prompts (the task) and the human-rating-based reward models. Both committee scorers are Qwen-family, distinct from the Llama-3.1-70B evaluation judge, and the generative model's references are real human reappraisals, never the held-out GPT-4 answers or the judge's outputs. Any pooled or synthetic training text is n-gram-checked against the held-out eval answers before use.

Intended use & limitations

Research artifact for the study of RLHF, reward modeling, and computational emotion regulation. It writes short cognitive reappraisals of negative interpersonal situations. It is not a clinical or mental-health tool and must not be used as one. Quality is validated only on the narrow reappraisal task and the evaluation above.

References

• —Reward committee used to train this policy: idealab-cs2/reappraisal-reward-model-v2 (seed0..2/ discriminative ensemble + grm/ generative member)
• —Li, J. Z., Herderich, A., Nair, P., & Goldenberg, A. (2025). Skill but not Effort Drive GPT Overperformance over Humans in Cognitive Reframing of Negative Scenarios. PsyArXiv. https://doi.org/10.31234/osf.io/fzvd8