overthelex/echr-livehrb-temporal-2k

echr-livehrb-temporal-2k

Temporally binned evaluation split for LiveHumanRightsBench (ECtHR human-rights judgment prediction). Two temporal contamination-control axes, built from verdict-free (contamination-controlled) ECtHR text.

• —regular_temporal (1000): ex-Ukraine cases from overthelex/echr-verdict-free, binned by decision year over 2017-2026 (100/bin), round-robin stratified by respondent country. Supports per-model pre/post training-cutoff analysis and temporal-drift plots (arXiv:2605.24452).
• —ukr_temporal (1000): from overthelex/echr-ukr-verdict-free, split at Russia's full-scale invasion (2022-02-24): 500 pre / 500 post, round-robin by ECHR article -> geopolitical-shift axis.

Natural violation/no-violation base rate preserved (base-rate balancing is a separate task). Public HUDOC sources only -> redistributable. Seed=42. Full judgments/decisions only (HUDOC doctype 001); Information Note summaries (002) excluded.

Columns

itemid, casename, respondent, article, violationlabel, verdictfreetext, decisiondate, applicationnumber, importance, ecli, **group** (regulartemporal | ukrtemporal), **bin** (year string | pre2022 | post_2022), plus length fields.

Usage

from datasets import loaddataset ds = loaddataset("overthelex/echr-livehrb-temporal-2k", split="train") # per-year drift on regular cases y2024 = ds.filter(lambda r: r["group"] == "regulartemporal" and r["bin"] == "2024") # Ukraine pre/post full-scale invasion ukrpre = ds.filter(lambda r: r["bin"] == "pre2022") ukrpost = ds.filter(lambda r: r["bin"] == "post_2022")

Changelog

v1.3 (2026-08-26) — verdict leak repaired

171 of 2000 rows (8.6%) still contained the Court's reasoning and have been re-cut. Verdict removal was marker-driven ("The Court's assessment" and similar); where no marker fired, the entire legal analysis survived into the field models are scored on. BALL v. ANDORRA retained 18,160 characters after THE LAW, including the conclusion, against a no_violation label.

The v1.1 leakage audit missed this because it was lexical. A lexical probe fails in both directions: it fires on domestic courts quoted in the facts ("the City Court finds no violation of Article 6") and misses the Court phrasing its own conclusion as "did not fail to fulfil its positive obligations". retention_percentage is not a usable filter either — affected rows appear at 49% and 66% retention.

The skew mattered more than the rate. Leakage was concentrated in exactly the dimensions this data is compared on: regular 18.5% against ukr 0.1%, no_violation 2.4–2.6x more often than violation, importance-1 judgments 21.3% against 4.9% for importance-4, and per-country rates from 52% to 0%. Measured impact on affected rows: no effect on violation cases, +14.3pp on no-violation accuracy, +7.1pp balanced.

What changed: text is now cut at the last THE LAW header following the last THE FACTS, ignoring table-of-contents repetitions. Only affected rows were touched — every other row is byte-identical to v1.2. Row membership is unchanged, so results from v1.2 remain comparable except where the text itself changed. verdict_free_length, retention_percentage and verdict_removal_method were recomputed to follow the text.

Reproduce the audit with scripts/leak_audit/audit_leak.py in the LiveHumanRightsBench repository.