riltonfranzone/legal-reward-bench

LegalRewardBench

LegalRewardBench accompanies Building Reward Models for Grounded Legal Reasoning. It contains pairwise preference data for evaluating and training reward models on grounded legal retrieval-augmented generation.

The primary benchmark is LegalRewardBench-v2.

Files

Use these files for the main benchmark:

data/legal_reward_bench_v2/train.jsonl
data/legal_reward_bench_v2/dev.jsonl
data/legal_reward_bench_v2/test.jsonl
data/legal_reward_bench_v2/valtest.jsonl

Use this file for the combined DPO training run described in the paper:

data/training/cjb_lrb_v2_train_dpo.jsonl

Other included artifacts:

data/
  legal_reward_bench/
    train.jsonl, dev.jsonl, test.jsonl
  context_variant_ablation/
    pairs.jsonl, strict_pair_yield.csv, answerable_summary.csv, unanswerable_summary.csv
  contextual_judge_bench/
    train.jsonl, test.jsonl
  training/
    legal_reward_bench_v2_train_truncated.jsonl, cjb_lrb_v2_train_dpo.jsonl
  transfer/
    *.jsonl
intermediate/
  triples/, responses/, labels/
manifest.json

valtest.jsonl is the concatenation of dev and test used by the paper evaluation scripts.

Counts

LegalRewardBench-v2:

Totals: train 940, dev 124, test 156, valtest 280.

Combined training file: 2,540 rows.

• —940 LegalRewardBench-v2 training rows
• —1,600 ContextualJudgeBench training rows

Context-variant ablation:

• —data/context_variant_ablation/pairs.jsonl: 2,597 strict preference pairs
• —intermediate/labels/pointwise_labels_all_context_variants__gpt_oss_120b.jsonl: 4,800 pointwise labels
• —intermediate/responses/: 48 response files, four generators across twelve context variants

Schema

Pair files are JSONL. Core fields:

{
  "prompt": "...",
  "chosen": "...",
  "rejected": "...",
  "split": "faithfulness",
  "triple_id": "...",
  "chosen_labels": {},
  "rejected_labels": {},
  "_modified": ""
}

split is the preference category, not train/dev/test membership. Dataset membership is determined by the filename.

Preference categories: refusal_answerable, refusal_unanswerable, faithfulness, correctness, completeness.

LegalRewardBench-v2 includes _modified for length-balanced replacements:

• —rejected_to_hallucination
• —chosen_to_elaborate_refusal
• —rejected_to_incomplete

The combined training file also includes source.

Loading

from datasets import load_dataset

lrb_v2 = load_dataset("riltonfranzone/legal-reward-bench", "legal_reward_bench_v2")
train = lrb_v2["train"]
dev = lrb_v2["dev"]
test = lrb_v2["test"]

Combined DPO training data:

from datasets import load_dataset

train = load_dataset("riltonfranzone/legal-reward-bench", "combined_training", split="train")

Intermediate Artifacts

The intermediate/ directory contains triples, generated responses, and GPT-OSS-120B pointwise labels used to build the preference pairs. Use data/ files for training and evaluation.

manifest.json records row counts, file sizes, and SHA-256 hashes.

Caveats

Pointwise labels are LLM-judge labels, not expert human legal annotations. The dataset is for research on grounded legal RAG, abstention, and reward-model evaluation. It should not be used as legal advice.

Citation

If you use this dataset, cite:

@misc{legalrewardbench2026,
  title = {Building Reward Models for Grounded Legal Reasoning},
  year = {2026},
  note = {LegalRewardBench dataset release}
}