anonymous-rankjudge-neurips/RankJudge

RankJudge

Anonymous submission. Under double-blind review at NeurIPS 2026 Datasets & Benchmarks. Authors and affiliations have been removed. The dataset will be re-released under the authors' names after the review period.

RankJudge is a benchmark for evaluating LLM-as-judge on multi-turn conversation quality, grounded in verifiable reference material. For each source item (an academic paper or financial filing with reference QA pairs), the pipeline generates a pair of conversations: a good one where the assistant is faithful to the source, and a bad one where it exhibits a single specific weakness in exactly one round. 21 LLM judges then evaluate every pair and predict the verdict, the bad round, and the weakness type. Because each pair has a known ground-truth answer, judge accuracy is objective.

Domains

• —ml — CS papers (open-access ML/CS preprints with QA pairs).
• —med — medical research papers (PubMedQA-style).
• —fin — S&P 500 10-K SEC filings.

Configs (no train/validation/test splits)

Each config is a single named split equal to its config name.

A non-tabular file metrics/metrics.json is also shipped in the repo (Bradley-Terry and RankLLM aggregates under 4 slices × 2 correctness criteria, broken down by weakness, user behavior, and domain). Load it with huggingface_hub.hf_hub_download.

Loading

from datasets import load_dataset

pairs    = load_dataset("anonymous-rankjudge-neurips/RankJudge", "pairs_filtered")
matches  = load_dataset("anonymous-rankjudge-neurips/RankJudge", "matches")
verifies = load_dataset("anonymous-rankjudge-neurips/RankJudge", "verification")

# Aggregate metrics (raw JSON, not tabular)
from huggingface_hub import hf_hub_download
import json
metrics = json.load(open(hf_hub_download(
    repo_id="anonymous-rankjudge-neurips/RankJudge",
    repo_type="dataset",
    filename="metrics/metrics.json",
)))

Schema highlights

pairs / pairs_filtered (one row per pair)

matches (one row per (pair, judge) prediction)

verification (one row per pair, joinable to pairs.id)

• —coherence.{good,bad}_ok + _issue — is the plan internally consistent?
• —adherence.{good,bad}_followed + bad_flaw_round_correct + _issue — did each conversation follow its plan, with the flaw landing in the right round?
• —grounding.{good,bad} — per-round, per-claim grounding judgments against the source. The flawed round is excluded from the bad conversation's grounded_rate.

Taxonomies

7 assistant weaknesses (metadata.assistant_behavior_type): disorganized, evasion, fabricated_answer, instruction_forgetting, no_clarification, self_contradiction, unnecessary_refusal.

7 user behaviors (metadata.user_behavior_type): exploratory, focused, integrative, misinformed, scattered, skeptical, underspecified.

Judge models

21 frontier and mid-tier LLMs are scored:

deepseek-v3.2, gemini-3-flash, gemini-3.1-pro, gemma-4-26b-a4b,
gemma-4-31b, glm-5.1, gpt-5.4, gpt-5.4-mini, gpt-5.5, gpt-oss-120b,
gpt-oss-20b, haiku-4.5, kimi-k2.6, minimax-m2.7, mistral-small-4-119b,
opus-4.7, qwen3.5-122b-a10b, qwen3.5-35b-a3b, qwen3.5-397b-a17b,
qwen3.6-plus, sonnet-4.6

metrics.json shape

metrics.json is a 5-level nested dict:

{criterion} -> {domain} -> {subset} -> {method} -> {judges: {judge: {elo, win_rate, elo_se, elo_ci95}}, pairs: {...}}

• —criterion ∈ {correct, verdict_and_round} — correct requires verdict + round + weakness; verdict_and_round drops the weakness requirement.
• —domain ∈ {all, ml, med, fin}.
• —subset ∈ {original, filtered, top_removed, partial_obs}. top_removed is the published evaluation slice: filtered minus pairs flagged non-ok by human auditors minus the top-Elo cut.
• —method ∈ {bradley_terry, by_assistant_behavior, by_user_behavior, by_domain}.

The headline numbers in the paper come from correct/<domain>/top_removed/bradley_terry.

License

Released under CC-BY-4.0. Source documents (PubMed abstracts, SEC 10-K filings, open-access CS papers) are public-domain or openly licensed; conversation pairs and judge predictions are generated artifacts.

Citation

@inproceedings{anonymous2026rankjudge,
  title  = {RankJudge: Elo-style Evaluation of LLM Judges on Multi-turn Conversations},
  author = {Anonymous},
  booktitle = {Submitted to NeurIPS 2026 Datasets \& Benchmarks Track},
  year   = {2026},
  note   = {Under review.}
}