arthrod/contractscrub-repro

ContractScrub reproduction (synthetic variant)

Reproduction of ContractScrub: A benchmark for final review of legal contracts (arXiv:2608.20204, Thomson Reuters Foundational Research). The official gold (tri-fair-lab/contract_scrub) was still unreleased at reproduction time, so this repo contains a method-faithful pipeline + a synthetic benchmark with ground truth true by construction — not the paper's leaderboard numbers.

Contents

• —SCORING.md — detailed walkthrough of the deterministic multiset scoring: parsing, normalization, location canonicalization, symmetric paired-location matching, multiset TP/FP/FN, macro recall/F1, and the term-only ablation
• —scrub/prompts.py — the 9 scrub categories + prompt assembly, reconstructed from the paper's Appendix G
• —scrub/eval_lib.py — deterministic multiset scoring: text normalization, location canonicalization (1(a)(i) -> 1ai), symmetric paired-location matching, macro precision/recall/F1, term-only ablation (paper Table 5)
• —scrub/make_synthetic.py — benchmark generator: 44 CUAD contracts (15k-60k chars, ~10-15 pages), errors injected in all 9 categories with gold logged at injection time (benchmark/benchmark.jsonl, 980 gold items)
• —scrub/run_eval.py — vLLM runner (chat-templated prompts, temperature 0.0, one prompt per contract x category per the paper's Sec 3.5 protocol)
• —results/ — full-run results: macro recall 0.472 / F1 0.238 (exact match) and macro recall 0.682 / F1 0.356 (term-only), per-category tables, raw outputs

Model

nvidia/Gemma-4-26B-A4B-NVFP4 — NVFP4 quantization of google/gemma-4-26B-A4B-it, the open-weight model the paper itself tested. vLLM 0.29.0, kvcachedtype bfloat16 (NVFP4 checkpoint requests FP8 KV cache, unsupported on SM86), flashinfer sampler disabled (no nvcc in image). Run: a100-large, 396 prompts, 11 min.

Headline result vs the paper

Paper (Gemma-4-26B, their gold): overall recall ~0.365, F1 ~0.326. Ours (same model family, synthetic gold): macro recall 0.472, F1 0.238 — the paper's core finding (contract scrubbing is far below general-capability expectations, F1 well under 0.5) reproduces. The per-category ordering does NOT reproduce; see the accompanying report — several of our categories are lexically conspicuous or underdetermined by construction, which the paper's lawyer-authored gold is not.

Caveats

• —Synthetic gold is regex/injection-derived; Defined Terms gold shares the lexical bias of the paper's easiest categories.
• —Incorrect-reference gold pairs a random existing section, which a model cannot in principle identify — that category's 0.000 recall is a benchmark artifact, reported as such.
• —44 contracts, 980 gold items vs the paper's 3,014; temperature 0.0 vs the paper's 0.6.