ocelotnima/lexbench

LexBench — Frontier LLMs on Canadian case law

LexBench benchmarks frontier language models on how faithfully they brief Canadian case law (facts, issue, decision, reasons, ratio), scored against human-annotated reference briefs and full decision texts.

Current board: 8 frontier models · 50 closed-book + 20 open-book cases · updated 16 Jul 2026.

Tracks

• —Closed-book — the model gets only the case name and must recall the brief from parametric knowledge (contamination-sensitive; treat with caution).
• —Open-book / temporal holdout — the model is given the full decision text from A2AJ and must brief it. Holdout decisions postdate model training cutoffs, making this track contamination-resistant.

Scoring

1. 1.Rubric score (primary) — HealthBench-style checklist grading: 12–20 atomic, point-weighted, case-specific criteria per case (authored by a strong model, graded by a pinned grader model). Score = weight met ÷ total weight.
2. 2.LLM-judge (secondary) — accuracy, completeness, and groundedness per section, graded by a judge from a family with no contestant on the board, blind to model identity.
3. 3.Cosine similarity (secondary) — sentence-embedding similarity to the reference.
4. 4.Hallucination safety — mean judge groundedness; format/refusal/truncation rates reported separately.
5. 5.Cost & latency per case.

All headline scores carry 95% bootstrap confidence intervals; ranking gaps are tested with a paired bootstrap and statistically indistinguishable gaps are flagged.

Submissions

Evaluations are run by the maintainer (models are pinned in ai_models.csv via OpenRouter, temperature 0). To request a model, open a discussion or a PR on the GitHub repo.

Links

• —Code & methodology: https://github.com/okelot/LLMBenchmarkForCCL
• —Mirror site: https://okelot.github.io/LLMBenchmarkForCCL/
• —Raw results: data.json
• —Data sources: A2AJ (open corpus of 191k+ Canadian decisions) and public case-brief wikis.