idealab-cs2/reappraisal-reward-model-v2

Reappraisal Reward Model v2 (3-seed ensemble)

Cognitive reappraisal is reinterpreting a negative situation to reduce the emotion it causes. Li, Herderich, Nair & Goldenberg (2025), *Skill but not Effort Drive GPT Overperformance over Humans in Cognitive Reframing of Negative Scenarios* collected reappraisals for 6 negative interpersonal scenarios from 611 people and from GPT-4-0314, and had human raters score each for effectiveness. This repo is part of a project that uses those ratings to train reward models of reappraisal effectiveness and to RL-finetune an open model that writes reappraisals as effective as possible — with GPT-4-0314, the AI reappraiser in the original study, as the bar to beat.

This model is the v2 reward model from that project: a three-seed ensemble of a multidimensional reward model finetuned from Qwen/Qwen3-1.7B. Given a scenario and a reappraisal of it, each ensemble member predicts the human effectiveness ratings; the training reward is the ensemble mean − 0.5·std (the spread term discourages the policy from exploiting scenarios where the members disagree). Together with grm/, it is the committee reward behind idealab-cs2/reappraisal-4b-grpo-committee.

This repo also hosts, under grm/, the generative reward model that pairs with this ensemble to form the committee reward behind the project's strongest policy (idealab-cs2/reappraisal-4b-grpo-committee). See The committee reward below.

Model Details

• —Base Model: Qwen/Qwen3-1.7B (sequence classification head, multidimensional)
• —Training Method: regression on z-normalized human ratings; 3 independent seeds (seed0/, seed1/, seed2/)
• —Training Reward: ensemble mean − 0.5·std
• —Training Data: pooled human ratings from Li et al. (2025), Studies 1 + 3 + 4 (OSF z9e48) — ~3.6k unique human reappraisals of the 6 scenarios, ~112k effectiveness ratings
• —Held-out pairwise accuracy: 0.940 (mean over the 3 seeds)
• —Trained By: ruggsea

Model Description

The v2 corpus pools the original vignette study with two later data collections by the same authors, about 3× the ratings behind the earlier Bradley-Terry (unpublished) and regression (unpublished) reward models. Ratings are z-normalized per (study, rater, dimension) to remove rater severity and cross-study scale differences before training. The split is by reappraisal, so no reappraisal appears in both train and validation.

GPT-authored reappraisals are held out of training, so the reward model never learns from the answers it is later used to score against; win-rate comparisons against GPT-4-0314 and the frontier baselines stay clean.

On held-out within-scenario preference pairs the ensemble reaches 0.940 pairwise accuracy, up from 0.877 for the earlier single reward model; using it as the GRPO reward lifts the trained policy's single-pass win-rate against GPT-4-0314 from 0.698 (previous reward model) to 0.806.

Use it as a training reward, not as a best-of-N reranker. The ensemble is a strong local reward signal for GRPO but a weak global ranker of diverse candidate reappraisals (best-of-16 selection by this model scored around chance); reranking is better served by the Bradley-Terry model (unpublished).

Usage

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "idealab-cs2/reappraisal-reward-model-v2"
tok = AutoTokenizer.from_pretrained(repo, subfolder="seed0")
seeds = [AutoModelForSequenceClassification.from_pretrained(repo, subfolder=f"seed{i}",
         torch_dtype=torch.bfloat16).eval() for i in range(3)]

PROMPT = ("Rate how effective this reappraisal is for the person in the scenario.\n\n"
          "SCENARIO:\n{vignette}\n\nREAPPRAISAL:\n{reap}")

def reward(vignette, reap):
    x = tok(PROMPT.format(vignette=vignette, reap=reap),
            return_tensors="pt", truncation=True, max_length=512)
    with torch.no_grad():
        z = torch.stack([m(**x).logits.mean(-1).squeeze() for m in seeds])  # per-seed score
    return (z.mean() - 0.5 * z.std()).item()   # ensemble mean − 0.5·std

The committee reward

The ensemble above, used on its own, is the reward for reappraisal-4b-grpo-rmv2-committee) (single-pass win-rate 0.806 vs GPT-4-0314). The project's strongest policy (reappraisal-4b-grpo-committee-committee), 0.872) is trained on a two-member committee: this discriminative ensemble and a generative reward model of a different mechanism, shipped in this repo under grm/.

The generative member is an RM-R1-style reasoning reward model (also Qwen3-1.7B). Rather than predicting a score, it acts as a judge: given the scenario and two reappraisals it reasons about effectiveness, empathy, novelty and specificity, then picks the better one. We turn that into a per-reappraisal score by how often it beats the median-effectiveness human reappraisal for the same scenario, over both A/B orders. Combining a discriminative scorer with a generative judge helps because their mistakes are less correlated than two seeds of the same model. An equal-weight mix generalizes better off-distribution, while an RM-heavy (0.7/0.3) mix does worse, so the diversity is what pays.

Within each scenario group both scorers are z-normalized, then combined w_rm·z_rm + w_grm·z_grm − λ·std (a disagreement guard, Coste et al. 2023; defaults w=0.5/0.5, λ=0.5). This snippet reproduces that reward exactly (minus the length/format gate applied during GRPO):

import torch, numpy as np, re
from transformers import (AutoModelForSequenceClassification,
                          AutoModelForCausalLM, AutoTokenizer)

repo = "idealab-cs2/reappraisal-reward-model-v2"

# --- discriminative member: the 3-seed ensemble (repo root: seed0/1/2) ---
rm_tok   = AutoTokenizer.from_pretrained(repo, subfolder="seed0")
rm_seeds = [AutoModelForSequenceClassification.from_pretrained(
                repo, subfolder=f"seed{i}", torch_dtype=torch.bfloat16).eval()
            for i in range(3)]
RM_PROMPT = ("Rate how effective this reappraisal is for the person in the scenario.\n\n"
             "SCENARIO:\n{v}\n\nREAPPRAISAL:\n{r}")

def rm_score(v, r):
    x = rm_tok(RM_PROMPT.format(v=v, r=r), return_tensors="pt",
               truncation=True, max_length=512)
    with torch.no_grad():
        z = torch.stack([m(**x).logits.mean(-1).squeeze() for m in rm_seeds])
    return (z.mean() - 0.5 * z.std()).item()          # ensemble composite

# --- generative member: the RM-R1 pairwise judge (repo subfolder: grm/) ---
grm_tok = AutoTokenizer.from_pretrained(repo, subfolder="grm")
grm     = AutoModelForCausalLM.from_pretrained(
              repo, subfolder="grm", torch_dtype=torch.bfloat16,
              device_map="auto").eval()
JUDGE_SYS = ("You judge cognitive reappraisals. Given a scenario and two reappraisals (A and B), "
             "decide which would be MORE EFFECTIVE at helping the person feel better. Reason about "
             "four qualities -- effectiveness, empathy, novelty of perspective, and specificity to "
             "the situation -- then end your reply with exactly 'ANSWER: A' or 'ANSWER: B'.")
JUDGE_USR = "SCENARIO:\n{v}\n\nREAPPRAISAL A:\n{a}\n\nREAPPRAISAL B:\n{b}\n\nWhich is more effective?"

def _pick(v, a, b):
    msgs = [{"role": "system", "content": JUDGE_SYS},
            {"role": "user",   "content": JUDGE_USR.format(v=v, a=a, b=b)}]
    ids = grm_tok.apply_chat_template(msgs, add_generation_prompt=True,
                                      return_tensors="pt").to(grm.device)
    out = grm.generate(ids, max_new_tokens=512, do_sample=False)
    txt = grm_tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True).upper()
    m = re.findall(r"ANSWER:\s*\(?([AB])\)?", txt)
    return m[-1] if m else None

def grm_score(v, cand, human_ref):
    # win-prob of the candidate over the median-effectiveness human reference, both orders
    return (float(_pick(v, cand, human_ref) == "A") +      # ours = A
            float(_pick(v, human_ref, cand) == "B")) / 2   # ours = B

# --- committee reward over a GROUP of candidates for one scenario ---
def _z(a):
    a = np.asarray(a, float); s = a.std()
    return (a - a.mean()) / s if s > 1e-6 else a * 0.0

def committee_reward(scenario, candidates, human_ref, w_rm=0.5, w_grm=0.5, lam=0.5):
    z_rm  = _z([rm_score(scenario, c)             for c in candidates])
    z_grm = _z([grm_score(scenario, c, human_ref) for c in candidates])
    combo = w_rm * z_rm + w_grm * z_grm
    guard = lam * np.std(np.stack([z_rm, z_grm]), axis=0)  # 2-member disagreement guard
    return combo - guard                                   # reward per candidate

human_ref is the median-effectiveness human reappraisal for that scenario, taken from the Li et al. ratings. It is never a GPT-4 or frontier answer, so the held-out eval opponents never enter the reward. Both committee members are Qwen-family, distinct from the Llama-3.1-70B evaluation judge.

References

• —Bradley-Terry reward model (better for reranking): reappraisal-bt-reward-model (unpublished)
• —Policy trained on this ensemble alone: reappraisal-4b-grpo-rmv2 (unpublished; superseded by the committee policy)
• —Strongest policy, trained on the committee (this ensemble + grm/): idealab-cs2/reappraisal-4b-grpo-committee
• —Li, J. Z., Herderich, A., Nair, P., & Goldenberg, A. (2025). Skill but not Effort Drive GPT Overperformance over Humans in Cognitive Reframing of Negative Scenarios. PsyArXiv. https://doi.org/10.31234/osf.io/fzvd8