Datasetpublic

zlyngkhoi/nllp-cuad-multihop

NLLP 2026 — Synthetic Multi-hop Legal QA (CUAD within-document) Synthetic multi-hop legal question-answering dataset generated over the CUAD (Contract Understanding Atticus Dataset) contracts. Built for RAG training and evaluation: every question requires combining a definition or obligation from one part of a contract with a clause/figure elsewhere in the same document. Key stats Verified multi-hop questions 3,992 (balanced across hops/difficulty)… See the full description on the dataset page: https://huggingface.co/datasets/zlyngkhoi/nllp-cuad-multihop.

sourceHugging Facecc-by-4.0updated 24d agoView on Hugging Face

0likes69downloads

Dataset Card

NLLP 2026 — Synthetic Multi-hop Legal QA (CUAD within-document)

Synthetic multi-hop legal question-answering dataset generated over the **CUAD** (Contract Understanding Atticus Dataset) contracts. Built for RAG training and evaluation: every question requires combining a definition or obligation from one part of a contract with a clause/figure elsewhere in the same document.

Key stats


Verified multi-hop questions	3,992 (balanced across hops/difficulty)
Source contracts	90 (CUAD, official train split)
Hop range	2–5
Difficulty cells	12 (`{2,3,4,5}hop × {easy,medium,hard}`)
Split (gold-chunk-disjoint)	train 3,218 / val 774

Hop distribution: 2-hop 101 · 3-hop 273 · 4-hop 1,676 · 5-hop 1,942.

Files

File	Contents
`dataset_grpo.jsonl`	all 3,992 samples, GRPO/training format
`dataset_train_grpo.jsonl` / `dataset_val_grpo.jsonl`	train / val split
`dataset_final.csv`	flat table (question, answer, hops, difficulty, gold_path)
`dataset_eval.jsonl`	self-contained eval rows (question + full document + chunks + gold answer)
`documents.jsonl`	the 90 source contracts (chunk-reassembled)

Sample format (`dataset_grpo.jsonl`)

json

{
  "question_id": "qa_path_2hop_32",
  "prompt": [{"role": "system", "content": "You are a contract review assistant ..."},
             {"role": "user", "content": "What is the length of a Contract Year ...?"}],
  "gold_answer": "twelve (12) months",
  "gold_path": ["<doc>::204", "<doc>::8", "<doc>::3"],
  "gold_passages": [{"chunk_id": "...", "text": "..."}],
  "hop_count": 2,
  "difficulty_cell": "2hop_hard",
  "answerable": true,
  "answerability_f1_relaxed": 1.0,
  "cuad_categories": ["Expiration Date"]
}

Methodology (why the questions are genuinely multi-hop)

—Definitional dependency: an earlier passage defines a term/rate/threshold/ deadline/party; the last passage applies it opaquely ("such rate", "as set forth in Section 4.1") or combines it with a figure. The answer is the defined value or a derived value that appears in no passage verbatim.
—Answer-first generation: the concrete answer is fixed first, then a question is written that requires traversing the full chain — never question-first.
—LLM verification (3 solver checks) — every accepted sample passed:
—Answerability: a solver given ALL gold passages reproduces the answer (relaxed F1 ≥ 0.5).
—Chain dependency: a solver given ONLY the last passage cannot answer (F1 < 0.7).
—Retrieval necessity: the single most-similar retrieved passage alone does not carry the answer.
—78.2% of generated questions passed; ~4,364 accepted in total, curated to a balanced 3,992.

Usage

GRPO / RL fine-tuning: feed prompt to the model; reward against gold_answer (exactness) plus a grounding check using gold_passages. RAG eval: use dataset_eval.jsonl (document + chunks are embedded per row, so any retriever/reader can be scored closed-context without extra corpora).