gt-csse/false-citation-bench

False Citation Bench

False Citation Bench is a compact evaluation and inspection dataset for false or misleading case citations in legal documents. It contains 26 source documents, their PDFs, and manually reviewed citation annotations grounded in the local text extraction.

Dataset contents

The repository has one matching document in each directory:

documents_txt/{index}__{case-name}__{filing}.txt
documents_pdf/{index}__{case-name}__{filing}.pdf
annotations/{index}__{case-name}__{filing}.json
derived/extraction.jsonl                           594 citation identifiers and their spans
derived/validation-courtlistener-heuristics.jsonl  423 occurrences CourtListener can decide

comparison_only__courtlistener_plain_text/{index}__{case-name}__{filing}.txt

comparison_only__courtlistener_plain_text/ is not part of the annotated corpus. It holds a second rendering of the same PDFs, used to justify one preprocessing choice and nothing else; no span in this dataset indexes into it. See its own note.

There are 79 annotated case-citation records. Other references in the source documents - statutes, rules, and non-conventional citations - are not annotated; the dataset's scope is case citations only. The numeric document_id remains inside each JSON record and the manifest; filenames use a zero-padded document index, normalized case name, and filing description for human readability. This artifact does not define train, validation, or test splits; it is intended for evaluation, error analysis, and annotation review.

What "derived" means here

For the purpose of evaluating the mellea-lrc project, we include two derived datasets with their own annotations and data cards:

• —`derived/extraction.jsonl` — 594 citation identifiers and their spans.
• —`derived/validation-courtlistener-heuristics.jsonl` — 423 occurrences CourtListener can decide.

Both are derived from the same 26 filings — not from the 79 annotations. They answer different questions.

The two derived sets nest: every one of the 423 validation occurrences is also an extraction record, and the remaining 171 extraction records are identifiers CourtListener cannot decide — a citation must be found before it can be checked.

annotations/ does not nest into either, because its unit is different. It records a judgement about cited authority, not a span to be detected, and most of the 594 extraction records are ordinary correct citations carrying no annotation at all. Only annotations/ holds the false-citation judgement, and it remains the primary artifact — a derived set is an evaluation target for one pipeline stage, not a second opinion on whether a citation is false.

The minimum sufficient case identifier

Every span in this dataset is keyed on one idea, so it is worth stating precisely.

The minimum sufficient case identifier of a citation is the shortest run of text that, by itself, picks out exactly one authority in a citation database.

A reporter locator is globally unique: no two cases share a volume, reporter and page. A docket number is not — 1:19-CV-362 exists in many districts — so its court is part of the identifier rather than payload attached to it.

Each is one span in the text, but the identifier is not the span. It is what the span resolves to: three parts for a locator, a number and a court for a docket. A system reports the identifier, and the span says which occurrence of it is meant.

Everything else a citation carries is corroborating, not identifying: case name, year, court parenthetical, pin cite. A citation can state each of them wrongly and still resolve, which is precisely the situation this dataset is about.

Why the datasets are keyed on it

It has determined edges. A citation's full extent does not. Where does Ashcroft v. Iqbal, 556 U.S. 662, 678 (2009) (citing Twombly) begin and end — at the case name, the parenthetical, the pin cite? Reasonable systems disagree, so a benchmark keyed on full extent penalises disagreement about boundaries rather than about citations. 556 U.S. 662 starts where the volume starts and ends where the page ends.

It is a complete assertion. Parsing a locator once it is delimited is trivial. A system that returns the exact span has thereby demonstrated it both found the citation and understood what it was, so exact-span agreement is sufficient for scoring and no further field needs to be compared.

It is stable across versions. Identifiers do not overlap in these documents, so one occurrence is one identifier at one place, unambiguously, across a dataset rebuild.

It is comparable across renderings. Two filings may write the same reporter as F.Supp.2d and F. Supp. 2d. Compared with punctuation and spacing removed, those are one identifier — so a system is scored on the authority it reported, not on which spelling the PDF happened to yield. This is why a record exists only when the text states its identifier completely: a fragment normalizes to something that is not an identifier at all.

Why it matters outside the benchmark

It is the unit of lookup. Verifying a citation begins by resolving the identifier — a reporter database for a locator, RECAP for a docket — and every later check depends on having done so. A system that cannot produce the identifier cannot start.

It is also the last thing to survive damage, and its loss is silent. Case names, parentheticals and pin cites can arrive mangled and a citation is still checkable; break the identifier and the citation does not degrade, it vanishes from every rule-based extractor at once, and nothing downstream reports a problem because nothing downstream was told there was a citation there. This is what makes preprocessing consequential rather than cosmetic, and it is the reason for the comparison in Text and span alignment below.

Finally, it is the minimum useful output. A tool that surfaces nothing but the identifier and its location has still given a reviewer everything needed to check the citation by hand. That is the floor this dataset measures against.

Annotation format

Each annotations/{id}.json file contains document metadata and an annotations array. Every annotation identifies:

• —citation_span: the cited case citation in the local text
• —proposition_span: the proposition associated with that citation
• —type: the citation problem category
• —annotation_source: how this record's problem determination was established. {"annotation_type": "court_ruling", "court_id": ..., "case_number": ..., "description": ...} for a citation a court itself flagged - these identify the court's own order/ruling (not the filing containing the false citation) by its docket identity and a one-line description of what it ruled on; every one has been verified against that ruling's own text or case summary to confirm it genuinely discusses this document's citations, not just a coincidental case-name/ECF match. {"annotation_type": "manual_check"} for a citation identified through independent verification rather than a court ruling.
• —problem: the project-specific problem label
• —reason: the supporting explanation
• —asserted_proposition: the proposition expressed by the filing

The two citation-problem types are:

• —unverifiable_authority: fabricated cases and wrong citations that cannot be attributed to a clear, verifiable authority
• —misrepresented_authority: a recognizable authority is cited, but the authority does not support the asserted proposition or quotation

The same citation may have multiple annotations when it is associated with distinct propositions.

Text and span alignment

All offsets and span text are computed against the corresponding file in documents_txt/, which contains the local Docling extraction of the PDF. Each PDF in documents_pdf/ is the corresponding source document from the reference corpus.

Why not the CourtListener text

CourtListener stores a plain_text rendering for most RECAP documents, including all 26 of these, and the reference corpus this dataset builds on carries spans measured against it. Those offsets are not reused here. The two preprocessing pipelines are not span-aligned: they are different renderings of the same PDF, and a character offset in one means nothing in the other.

The choice of which rendering to annotate is not arbitrary. Layout-aware PDF extraction breaks fewer citations, and we measured the difference on this corpus by running the same eyecite-based extractor over both renderings of the same 26 filings:

The gap is not random loss. Of the 56 locators the CourtListener text costs, 46 are still present in it — they reappear once whitespace is collapsed — and only 10 are absent in any form. The identifier is there; a line break has been inserted into the middle of it, and a citation parser joining volume, reporter and page with a literal space can no longer see it.

Both renderings ship, so this is reproducible rather than asserted: the CourtListener text is under `comparison_only__courtlistener_plain_text/`, with the script that produces the table above. Those files exist for this comparison alone — they are unannotated, their captions are unmasked, and no offset in this dataset refers to them.

That is why we expect layout-aware preprocessing to serve most implementations better: it is markedly less likely to break the minimum sufficient case identifier, and breaking that identifier is not a degradation but a disappearance.

One caveat on the comparison. The gold set is defined on the Docling rendering, so a citation that Docling mangled and CourtListener preserved could not enter it. The four hand-added records in derived/extraction.md are the known cases of Docling damage, and they are counted as losses for Docling above.

Why these documents are redistributable

Every document here is a party filing entered on a public federal court docket and obtained through PACER via RECAP, where the same filings are already freely available. They are republished here so that results computed on them can be reproduced.

Intended use and limitations

Use this dataset to inspect citation-grounding failures, evaluate citation-verification systems, and develop legal-document error analysis tools. The annotations describe citation problems identified in the source records; they are not a substitute for independent legal research or legal advice.

Preprocessing applied to documents_txt/

These are Docling extractions of the PDFs, with one deliberate edit: the eleven places where a filing states its own docket number — eight case captions, a letter header, a PACER footer stamp, and one in-body reference to the filing's own matter — are blanked with spaces of equal length, so no offset shifts and every derived span stays valid.

They are removed rather than filtered at evaluation time because a caption is not a citation to authority, and nothing a system reports at one should count for or against it. Left in, those positions move a score for reasons unrelated to the task: a system is penalised for finding a docket number that is really there, or rewarded for suppressing it. Filtering instead would make every consumer reimplement the same discrimination, and bake a caption classifier into the pipeline — a step that can lose or distort information no one asked it to touch. The deployment case is a lawyer checking a draft, which has no caption at all, so such a classifier would fit an artifact of collecting filed documents rather than the work.

The eleven docket numbers that remain all cite other cases. documents_pdf/ is untouched and remains the source of record.