overthelex/echr-ukr-verdict-free

ECtHR Ukraine Verdict-Free Dataset

2,619 verdict-free case-article pairs from the European Court of Human Rights (ECtHR) involving Ukraine as respondent state. Each record contains the factual background, procedural history, parties' arguments, and cited legal standards -- with the Court's legal reasoning, findings, conclusions, and operative provisions surgically removed.

Dataset Description

This dataset is designed for evaluating LLM reliability in legal judgment prediction. By removing the Court's verdict from each case, models must predict violation outcomes based solely on pre-decisional material -- the same information available to parties before judgment.

Key Statistics

Verdict Removal Pipeline

Three-stage pipeline ensures no verdict leakage:

1. 1.Pattern-based truncation -- removes text after markers introducing the Court's assessment (e.g., "THE COURT'S ASSESSMENT", "MERITS", "FOR THESE REASONS", "OPERATIVE PROVISIONS")
2. 2.Dual LLM verification -- Claude Sonnet 4.6 and Claude Haiku 4.5 independently verify no verdict leakage remains. Cross-family verification with Amazon Nova Pro and Qwen3 32B
3. 3.Repair or exclude -- cases with detected leakage are repaired by removing leaking sentences; cases that become too short are excluded
4. 4.Post-verification regex scan -- iterative pattern-based cleanup of any remaining conclusory language

Articles Covered

Dataset Structure

Each record contains:

• —item_id -- HUDOC item identifier
• —case_name -- case title
• —article -- ECHR Convention article number
• —violation_label -- ground truth: "violation" or "no_violation"
• —verdict_free_text -- full case text with verdict removed
• —original_length -- character count of original text
• —verdict_free_length -- character count after verdict removal
• —retention_percentage -- percentage of text retained
• —verdict_removal_method -- method used for verdict removal
• —article_full -- protocol-aware ECHR code (e.g. P1-1); falls back to article when unresolved

Source

Cases sourced from the HUDOC database (hudoc.echr.coe.int) of the European Court of Human Rights. Only English-language judgments with Ukraine as respondent state are included.

Changelog

v1.2 (2026-07-14) — added article_full:

• —New non-breaking column article_full with protocol-aware ECHR codes (e.g. P1-1, P4-2, P7-4), recovered by re-parsing the HUDOC conclusion. The legacy article field collapsed protocol prefixes (article == "1" is Article 1 of Protocol 1 / property, not Convention Article 1; article == "4" conflates Convention Art 4 and Protocol 4). article is unchanged for continuity; rows that cannot be resolved fall back to it. 221 rows carry a protocol code. Key per-article tests on `article_full`.

v1.1 (2026-07-03) — leakage audit and fixes (same pass as the livehrb-2k eval sets):

• —Conclusion scrub: verdict-revealing sentences removed: 1,423 sentences across 1,383 rows (73 rows / 2.8% leaked the gold label for their own target article) (patterns like "there has been (a|no) violation of Article X", "the Court finds/concludes that there has been...", "unanimously, that..."). Affected rows carry a +conclusion_scrub suffix in verdict_removal_method; verdict_free_length / retention_percentage recomputed.
• —respondent column added (constant UKRAINE) for schema parity with echr-verdict-free.
• —Missing decision_date values backfilled from the HUDOC API (100% coverage now).

v1.0 — initial release.

Usage

from datasets import load_dataset
ds = load_dataset("overthelex/echr-ukr-verdict-free")

Related Work

• —LiveLegalBench (Jin, Zhang et al.) -- perturbation framework for legal judgment stability
• —Multi-Legal-Bench -- cross-jurisdictional legal reasoning evaluation
• —UA-Legal-Bench -- Ukrainian legal reasoning benchmark

Citation

@misc{ovcharov2026echrukr,
  title={ECtHR Ukraine Verdict-Free Dataset},
  author={Ovcharov, Volodymyr},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/overthelex/echr-ukr-verdict-free}
}

License

CC-BY-SA-4.0