Datasetpublic

openbenchmarks/OB-LegalQA

OB LegalQA 95 questions on 7 redlined contract PDFs. They evaluate document parsers, not models: can a parser give a downstream agent what it needs to answer a question about a heavily negotiated contract? Columns column meaning id stable item id question the question put to the agent answer gold answer, from the position the parties agreed answer_detail why that is the answer task_type which redline pattern the question turns on documents… See the full description on the dataset page: https://huggingface.co/datasets/openbenchmarks/OB-LegalQA.

sourceHugging Facecc-by-4.0updated 9d agoView on Hugging Face

0likes213downloads

Dataset Card

OB LegalQA

95 questions on 7 redlined contract PDFs. They evaluate document parsers, not models: can a parser give a downstream agent what it needs to answer a question about a heavily negotiated contract?

Columns

column	meaning
`id`	stable item id
`question`	the question put to the agent
`answer`	gold answer, from the position the parties agreed
`answer_detail`	why that is the answer
`task_type`	which redline pattern the question turns on
`documents`	contract id(s) the question is about
`pdf`	path(s) to the digital PDF in this repo
`pdf_scanned`	path(s) to the same contract as page images, no text layer
`match_correct`	strings that indicate the agent answered correctly
`match_stale`	strings that indicate it answered from struck text
`match_fused`	strings that indicate it merged struck and surviving text

views/ carries each contract as markdown, once with the deletions kept as ~~struck~~ and once with every edit unmarked.

Provenance

Contracts come from crosbylegal/RedlineBench (CC-BY-4.0), converted to PDF. Derived from RedlineBench by Crosby Legal, adapted. Licensed CC-BY-4.0.

Benchmark design: openbenchmarks.com/doc-processing.