overthelex/echr-livehrb-temporal-1k

echr-livehrb-temporal-1k

Temporally distributed, outcome-balanced evaluation set for LiveHumanRightsBench. 1,212 ECtHR case–article instances drawn from 976 judgments, verdict removed.

Why this set exists

The earlier releases (echr-livehrb-static-2k, echr-livehrb-temporal-2k) preserve the Court's natural base rate, which is 83.7% violation. That leaves only 327 no-violation cases in 2,000 — and measurement shows the interesting behaviour lives almost entirely in that class. It also turned out that marker-based verdict removal left the Court's reasoning intact in ~9% of rows, concentrated in exactly the rare class (19.3% of no-violation rows against 7.4% of violation rows).

This set is built to remove both problems: balanced on outcome, evenly spread over time, and cut at a structural boundary rather than by marker matching.

Construction

Source: HUDOC full-text judgments (001- identifiers), English, 2012–2026.

Selection, in order — filters are applied before the per-year quota, so that filtering cannot punch holes in the temporal design:

1. 1.English Chamber/Committee judgments (CASE OF …), 25k–90k characters.
2. 2.A single unambiguous outcome parsed from the HUDOC conclusion field; cases mixing violation and no-violation findings are excluded.
3. 3.The judgment must carry paragraph back-references, so that per-paragraph materiality analysis is possible.
4. 4.Verdict removed at the last THE LAW header following the last THE FACTS, ignoring table-of-contents repetitions.
5. 5.Rejected: no structural boundary (34), verdict-free text under 5,000 characters (20), cut removed nothing (17), reasoning still present after the cut (1).
6. 6.Quota: 67 judgments per year, up to 33 no-violation, the remainder violation.
7. 7.Each judgment is expanded into one instance per Convention article named in its conclusion. 19.6% of judgments touch more than one article, so a case-level row would leave the prompt's "a violation of Article X" undefined for them. Labels never conflict within a judgment because mixed-direction cases are excluded at step 2.

Deterministic throughout: ordering is by md5(item_id).

Composition

Two years fall short of a 33/34 label split because the pool is exhausted: 2019 (27 no-violation available) and 2026 (12, partial year).

Fields

instance_id (item_id|article_full), item_id, case_name, application_number, respondent, decision_date, year, importance, `article_full` (protocol-aware: 6, 8, P1-1, P4-2 — key every per-article analysis on this), article (legacy, lossy: protocol prefixes collapsed, so 1 means Article 1 of Protocol 1), violation_label (gold), is_procedural, conclusion, verdict_free_text (input), original_length, verdict_free_length, retention_percentage, verdict_removal_method.

Because instances from one judgment share a verdict_free_text, summarisation, fact extraction and entailment are computed once per item_id and reused across its instances.

The full judgment text is deliberately not published: it contains the verdict. Every row is recoverable from HUDOC by item_id.

Caveats

• —Respondent states are unevenly represented (Russia 123 of 976 judgments), reflecting the Court's docket rather than sampling.
• —Instances per year vary (74–91) even though judgments per year do not, because multi-article cases are not evenly distributed over time. The temporal quota is defined on judgments.
• —92.3% of judgments yield a paragraph-level materiality label from the Court's own back-references; the remainder cite only statute text or nothing resolvable. Yield runs 82% in 2014 to 99% in 2023, so the labelled subset is slightly newer than the set.
• —Importance is skewed to routine cases (494 at level 3, 443 at level 4, 29 at level 1).
• —Verdict removal is checked structurally, not by a language model. Reproduce with scripts/leak_audit/audit_leak.py in the LiveHumanRightsBench repository.
• —Recency does not prove absence from a model's training data.

Summaries (summaries/)

The RQ1 and RQ2 arms score models on a summary instead of the full judgment. Those summaries are published here because otherwise those two arms cannot be reproduced or audited from this repository — only the full-text arms could be.

One summariser writes for all eight judges on purpose. When each model summarised for itself, a drop in accuracy could equally mean "summaries lose material facts" or "this model writes bad summaries", and the model axis was not clean.

Rejected samples

A summariser reading verdict-free text can still recognise the case and supply the outcome from memory. The first build produced 39 summaries stating a conclusion that appears nowhere in their source — "Consequently the Court found no violation of Article 8" against text whose operative part had been cut out. Those hand the answer to the arm they feed.

They track the label: of the 51 summaries mentioning an outcome, 41 agreed with ground truth against a base rate near 55%. The 12 whose source also discusses an outcome are reportage — a domestic finding, or a prior Strasbourg judgment about the same applicant — and were kept.

All 39 were rejected and redrawn from the same summariser under the same prompt. The published file contains zero remaining recalled verdicts under the same detector (experiments/summaries.py:asserts_outcome).

This is worth stating plainly: a contamination check that compares accuracy either side of the training cutoff cannot see this. The leak is in what the summariser writes, not in how the judge scores.

Prompt

Summarize the following legal case to approximately 500 words, focusing on the key
facts relevant to the alleged violation(s).

Case Name: {case_name}
Full Text:
{full_text}

Summary (approximately 500 words):