overthelex/echr-livehrb-static-2k

echr-livehrb-static-2k

Balanced static evaluation set for LiveHumanRightsBench (ECtHR human-rights judgment prediction).

• —regular: 1000 pairs from overthelex/echr-verdict-free (Ukraine excluded), round-robin stratified by respondent country.
• —ukr: 1000 pairs from overthelex/echr-ukr-verdict-free, round-robin stratified by ECHR article.
• —Natural violation/no-violation distribution (base-rate balancing is a separate task). Seed=42, reproducible.
• —Verdict-free text (contamination-controlled). decisiondate / applicationnumber / importance / ecli attached from HUDOC (~94-98% coverage).
• —Full judgments/decisions only (HUDOC doctype 001); Information Note summaries (002) excluded.

Columns

itemid, casename, respondent, article, violationlabel, verdictfreetext, decisiondate, application_number, importance, ecli, group (+ length fields).

Usage

from datasets import loaddataset ds = loaddataset("overthelex/echr-livehrb-static-2k", split="train") regular = ds.filter(lambda r: r["group"] == "regular") ukr = ds.filter(lambda r: r["group"] == "ukr")

Changelog

v1.3 (2026-08-26) — verdict leak repaired

186 of 2000 rows (9.3%) still contained the Court's reasoning and have been re-cut. Verdict removal was marker-driven ("The Court's assessment" and similar); where no marker fired, the entire legal analysis survived into the field models are scored on. BALL v. ANDORRA retained 18,160 characters after THE LAW, including the conclusion, against a no_violation label.

The v1.1 leakage audit missed this because it was lexical. A lexical probe fails in both directions: it fires on domestic courts quoted in the facts ("the City Court finds no violation of Article 6") and misses the Court phrasing its own conclusion as "did not fail to fulfil its positive obligations". retention_percentage is not a usable filter either — affected rows appear at 49% and 66% retention.

The skew mattered more than the rate. Leakage was concentrated in exactly the dimensions this data is compared on: regular 18.5% against ukr 0.1%, no_violation 2.4–2.6x more often than violation, importance-1 judgments 21.3% against 4.9% for importance-4, and per-country rates from 52% to 0%. Measured impact on affected rows: no effect on violation cases, +14.3pp on no-violation accuracy, +7.1pp balanced.

What changed: text is now cut at the last THE LAW header following the last THE FACTS, ignoring table-of-contents repetitions. Only affected rows were touched — every other row is byte-identical to v1.2. Row membership is unchanged, so results from v1.2 remain comparable except where the text itself changed. verdict_free_length, retention_percentage and verdict_removal_method were recomputed to follow the text.

Reproduce the audit with scripts/leak_audit/audit_leak.py in the LiveHumanRightsBench repository.