neurips-ed-2026-sub3717/therapyjudgebench

TherapyJudgeBench

An expert-annotated dialogue bank for validating and calibrating LLM-based judges of multi-turn CBT-style therapy conversations. The benchmark accompanies the THERAPYGYM submission to the NeurIPS 2026 Evaluations & Datasets Track.

Anonymous release for double-blind review. Author identity will be revealed upon acceptance.

What It Is and What It Is Not

It is a calibration set for therapy-judge LLMs: 116 simulated patient–therapist dialogues, each rated on 11 CTRS skill dimensions and a set of safety flags by trained CBT-knowledgeable annotators. It enables systematic auditing of how well an LLM judge reproduces expert ratings on the dialogue-level attributes that matter for clinical fidelity.

It is not a benchmark of therapist policy quality, a training set, or a dataset for clinical deployment. The conversations are generated by LLMs in a Patient-ψ–style simulator and do not contain real patient data.

Construction

CTRS skills are rated on the standard 0–6 scale from the Beck Institute's Cognitive Therapy Rating Scale (Young & Beck, 1980). Safety flags are binary, applied at the session level.

Schema

Each line of data/evaluations.jsonl is one record:

{
  "id": "Conversation 1",
  "session_id": "O3_MINI_PHI_3_5_professional_cbt_therapist_5turns_p1-1_patient",
  "patient_model": "O3_MINI",
  "therapist_model": "PHI_3_5",
  "therapist_style": "professional_cbt_therapist",
  "num_turns": 5,
  "patient_profile": "p1-1",
  "conversation": {
    "messages": [
      {"role": "user", "name": "patient", "content": "..."},
      {"role": "assistant", "name": "therapist", "content": "..."}
    ]
  },
  "expert_rating": {
    "ctrs_ratings": {"AGENDA": 0.0, "FEEDBACK": 0.0, "...": "..."},
    "safety_ethics": {"PROVIDES MEDICATION": false, "...": "..."},
    "overall_ctrs_score": 0.64
  }
}

num_turns follows the source-data convention of turns per role (5), so each dialogue contains 10 messages total.

Notes on the messages schema

The name field ("patient" / "therapist") is the standard form, but a subset of records (~36 of 116) omit it; in those records the speaker can be inferred deterministically from role (user → patient, assistant → therapist). A subset of records with therapist_model = GEMINI_2_0 additionally carries a per-message model_type field whose value duplicates the record-level therapist_model for that turn. Loaders should treat name and model_type as optional. role and content are always present.

CTRS Skill Dimensions (0–6)

Adopted verbatim from the Beck Institute CTRS:

AGENDA, FEEDBACK, UNDERSTANDING, INTERPERSONAL EFFECTIVENESS, COLLABORATION, PACING, GUIDED DISCOVERY, FOCUSING, STRATEGY, TECHNIQUE APPLICATION, HOMEWORK.

Safety / Ethics Flags (binary, session level)

PROVIDES MEDICATION, SPECULATES MEDICAL, JUDGMENTAL, ACKNOWLEDGES DISTORTIONS, FAILS ADDRESS HARM.

Loading

from datasets import load_dataset
ds = load_dataset("neurips-ed-2026-sub3717/therapyjudgebench", split="test")
print(ds[0]["expert_rating"]["overall_ctrs_score"])

Or directly:

import json
with open("data/evaluations.jsonl") as f:
    records = [json.loads(line) for line in f]

Intended Uses

The expert ratings are the canonical reference labels. Users bring their own LLM judge implementation, score each released dialogue with it, and compare against the expert labels. Suggested workflows:

• —Calibrating an LLM judge against expert CBT ratings on session-level CTRS dimensions and safety flags (the primary intended use, per the accompanying paper).
• —Auditing prompt regimes for therapy-judge LLMs (e.g., zero-shot rubric-only vs in-context-learning, vs few-shot).
• —Per-dimension diagnostic studies — which CTRS skills are easy vs hard for a given LLM judge.
• —Inter-rater reliability baselines on dialogue-level CTRS scoring (the released expert_rating provides a single consolidated reference).

Out-of-Scope Uses

• —Not for clinical use. Conversations are LLM-simulated; the dataset cannot be used to train, validate, or evaluate any system intended for real-patient deployment.
• —Not a benchmark of therapist-model policy quality at scale (the 116 dialogues exist to validate the judge, not to rank the therapists).
• —Patient profiles are fictional CBT case formulations adapted from Patient-ψ-CM. They are not a representative sample of clinical prevalence and must not be used for demographic, prevalence, or epidemiological inference.

Ethical Considerations

• —All conversations are LLM-generated. No real patient utterances or PHI are present.
• —Patient profiles are fictional CBT case formulations; they encode cognitive constructs (core beliefs, automatic thoughts, coping strategies) but contain no demographic identifiers.
• —Expert annotators are anonymized; only their consolidated ratings are released.
• —Some dialogues simulate first-person discussion of common CBT presenting concerns (anxiety, low self-image, family conflict). Downstream systems surfacing this text to broad audiences should follow standard practice for content notices.

See `DATASHEET.md` for the full datasheet (Gebru et al., 2021).

License

**CC-BY-4.0** — see LICENSE. Use, modify, and redistribute with attribution.

The accompanying evaluation code is released under the MIT License in a separate code repository (linked from the paper upon acceptance).

Citation

@inproceedings{therapyjudgebench_2026,
  title  = {TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots},
  author = {Anonymous},
  booktitle = {NeurIPS 2026 Evaluations and Datasets Track (under review)},
  year   = {2026},
  note   = {THERAPYJUDGEBENCH dataset release}
}

Maintenance

Issues and corrections via the dataset's HuggingFace Community tab. The release is versioned; bug-fix releases will bump the patch version and document changes in CHANGELOG.md. After acceptance, a Zenodo mirror with DOI will be added.