anonymousNeurIPS2026submission4281/reviewarena-eval

ReviewArena-Eval

ReviewArena-Eval is the benchmark slice that accompanies the NeurIPS Evaluations & Datasets submission ReviewArena: A Large-Scale Cross-Conference Dataset and Benchmark for LLM Peer Review. It evaluates LLMs on the LLM-as-reviewer task: given a full paper and the structured review form used at that paper's venue-year, the model must produce overall rating, confidence, sub-scores, and free-text fields that are then compared against the actual human reviews on OpenReview.

The benchmark intentionally measures agreement with the historical review process, not objective paper quality. Human reviews are noisy, but they are the relevant target for systems that claim to approximate or assist human reviewing.

from datasets import load_dataset
ds = load_dataset("anonymousNeurIPS2026submission4281/reviewarena-eval", split="train")
print(ds)
# Dataset({features: [...], num_rows: 1002})

Composition

• —1,002 papers total
• —167 papers per numerically rated venue: NeurIPS, ICLR, ICML, CoRL, COLM, EMNLP
• —Within a venue, papers are sampled uniformly across released years
• —TMLR is excluded (no numeric overall score) but remains in the parent dataset for claim-evidence and discussion tasks

Sub-score axes also vary by venue-year — NeurIPS 2025 uses quality / clarity / significance / originality; earlier NeurIPS and recent ICLR use soundness / presentation / contribution; CoRL adds robotics-specific axes. A generic "rate from 1–10" prompt is invalid for half the benchmark, so prompts must be rendered per (venue, year) with the correct scale endpoints, field names, and sub-score axes.

Task

For paper p from venue-year (v, y) and template T_{v,y} matching that venue's review form, model M produces

M(T_{v,y}, p) → (overall_rating, confidence, sub_scores, text_fields)

Targets are the corresponding fields in the human reviews on OpenReview. Multiple human reviewers per paper are aggregated as the per-paper mean for numeric metrics and concatenated for text metrics.

Schema

Each row is one paper. Inherits the parent ReviewArena schema; the most relevant fields:

The review union schema (fields populated per venue) is documented on the parent dataset card.

Evaluation Protocol

• —PDFs → markdown with `nvidia/nemotron-ocr-v2`, truncated to 50,000 characters when needed. Extraction is fixed across models so differences are not confounded with OCR differences.
• —Inference at temperature 0 through Fireworks AI (or any OpenAI-compatible provider).
• —Prompts are venue-year-aware: native scale endpoints, field names, sub-score axes, and accept/reject threshold all rendered from (v, y). The template is a correctness condition, not prompt engineering — if the venue asked humans for a 1–4 recommendation, the model is asked for a 1–4 recommendation.

Metrics

Per-venue MAE values are not comparable across venues without the scale row — a CoRL error of 0.65 on a 1–4 scale is not the same as a NeurIPS error of 0.65 on a 1–10 scale.

Baseline Results

Six open-weight LLMs served through Fireworks AI at temperature=0, single-shot per paper.

Per-venue overall-rating MAE (signed bias)

Cross-axis summary

Key findings

1. 1.Best overall-rating MAE is 0.70 (Qwen3.6-Plus); the next four models cluster within 0.14 pooled MAE.
2. 2.Score compression is the dominant failure mode. On 1–10 venues, most models avoid ratings below 4 or above 7; on 1–4 venues they top out at 3/4. This produces positive bias on NeurIPS/ICLR (whose human distributions sit below the apparent model prior) and weak accept/reject separation.
3. 3.Text similarity ≠ reviewer reliability. Gemma-4-26B ties for the best BERTScore-F1 but accepts nearly every ICLR paper (macro-F1 0.394). Conversely, Qwen3.6-Plus has the best numeric agreement but is not the best text mimic.
4. 4.Sub-scores and overall rating come apart. GPT-OSS-120B is competitive on overall ratings but weaker on sub-scores; Kimi K2.6 better tracks auxiliary axes than overall recommendation.
5. 5.GLM-5.1 alone shows usable accept/reject discrimination (1.40-point gap on ICLR), at the cost of slightly worse pooled MAE.

The benchmark therefore stress-tests three properties: (i) numeric calibration on the venue's native scale, (ii) ranking-style discrimination between accepts and rejects, and (iii) free-text agreement — and shows that current open-weight LLMs are not reliable standalone reviewers, even though they often produce plausible-looking review text.

Suggested uses

• —Stress-test new LLMs (open or closed) as scientific reviewers
• —Fine-tune / RLHF reviewer models against held-out venue-year cells
• —Calibration research: rating compression, accept/reject discrimination
• —Studying mismatch between text similarity and numeric agreement
• —Few-shot / RAG benchmarks that need full paper text + human reviews

Relationship to the parent dataset

This benchmark is a fixed, reproducible subset of the parent ReviewArena corpus (51,529 papers, 196,099 reviews, 558,785 OCR pages). The parent dataset retains all venues including TMLR for claim-evidence and discussion tasks; this card covers only the 1,002-paper benchmark slice with venue-year-aware prompts.

Citation

Under double-blind review for NeurIPS 2026 Evaluations & Datasets. Please cite the camera-ready proceedings entry once public; until then use an anonymous placeholder consistent with your venue's reviewer guidelines.