neurips-ed-2026-submission-2998/LegalReasoningBench

LegalReasoningBench

A benchmark of structured gold-standard annotations for European Court of Human Rights (ECtHR) Article 11 (freedom of assembly) cases — covering case-level conclusions, six doctrinal element conclusions, and fact-to-element mappings (List 1 / List 2 / List 3).

Released alongside an anonymous NeurIPS 2026 submission.

Quick start

from datasets import load_dataset

ds = load_dataset("neurips-ed-2026-submission-2998/LegalReasoningBench", split="test")
print(ds[0])

The dataset has 375 examples, one per case.

Schema

{
  "case_name": "case_274",
  "atomic_facts": [
    {"fact_id": "F001", "text": "...", "source_paragraph": 3, "section_path": ["THE FACTS"]},
    ...
  ],
  "conclusion": {
    "violated": true,
    "elements": {
      "scope":             {"conclusion": "yes",  "discussed": "explicit"},
      "interference":      {"conclusion": "yes",  "discussed": "explicit"},
      "prescribed_by_law": {"conclusion": "yes",  "discussed": "implicit"},
      "legitimate_aim":    {"conclusion": "yes",  "discussed": "explicit",
                            "aims": ["prevention_of_disorder", ...]},
      "necessity":         {"conclusion": "no",   "discussed": "explicit"},
      "positive_obligation": {"conclusion": null, "discussed": "not_discussed"}
    }
  },
  "fact_mapping": {
    "list1": {"scope": ["F003", "F012", ...], "interference": [...], ...},
    "list2": {"scope": [...], ...},
    "list3": ["F005", "F012", ...]
  }
}

• —atomic_facts — structured facts extracted from the judgment (model input).
• —conclusion.violated — case-level Article 11 violation finding.
• —conclusion.elements[el].conclusion — "yes" / "no" / null (not discussed).
• —conclusion.elements[el].discussed — "explicit" / "implied" / "implicit" / "not_discussed".
• —fact_mapping.listN — fact IDs referencing atomic_facts[*].fact_id. Subset constraint: List 3 ⊆ List 2 ⊆ List 1.

Files

• —cases.jsonl — primary file, one case per line.
• —raw/ — supporting per-case JSONs (also used to regenerate cases.jsonl):
• —compiled_gold_standard/ — gold annotations (one file per case).
• —atomic_facts/ — atomic facts (one file per case).
• —included_all_case_names.txt — list of case IDs.

License

The annotations are released under CC BY 4.0. The underlying ECtHR judgment texts are public-domain documents from HUDOC.