Ihteshamstar/qwen3-4b-cuad-extractor
Qwen3-4B CUAD Contract Clause Extractor
Qwen3-4B-Instruct fine-tuned with QLoRA to extract all 41 CUAD clause types from a contract in one pass, quoting the contract text verbatim and returning explicit empty lists for absent clauses. Runs in ~2.5 GB (q4KM GGUF) on consumer hardware.
Trained and evaluated on the official CUAD split: 408 training / 102 held-out test contracts, no contract overlap.
Training and evaluation code, raw eval outputs, and build notes: github.com/Ihtesham-star/cuad_llm_finetuning
Results
Detection F1: did the model find each clause type present in a contract (and produce a verbatim quote for it)? Scored per (contract, clause type) over 102 test contracts × 41 clause types; predictions merged across contract chunks. Bootstrap 95% CIs over contracts (n=1000).
Three seeds (42/43/44) under identical config: F1 0.8995 ± 0.0017. The published weights are seed 42.
Strict full-span coverage (a true positive requires the predicted quotes to fully contain every gold span for the clause — our reimplementation of the rule used by ContractEval, arXiv:2508.03080, which benchmarked zero-shot LLMs on this same 102-contract test set):
Caveats for that comparison: our matching is a reimplementation of their stated rule (whitespace-normalized, casefolded substring), not their harness; and ContractEval queries one clause type per call with its definition in the prompt, while this model answers all 41 clause types in a single call (~40× fewer calls per contract).
Data curve (seed 42, detection F1 vs number of training contracts): 50 → 0.746 · 150 → 0.862 · 300 → 0.885 · 408 → 0.900. The zero-shot 14B baseline (0.816) is passed somewhere between 50 and 150 annotated contracts.
Prompt format
The model was trained on exactly one prompt shape. Use it verbatim; other phrasings are out of distribution.
System:
You are a contract review system. Extract the requested clause types from the contract excerpt. Quote the contract text verbatim. A clause type not present in this excerpt gets an empty list. Reply with JSON only.User:
Extract all 41 CUAD clause types from this contract excerpt as JSON (keys: clause types, values: lists of verbatim quotes).
CONTRACT EXCERPT:
<up to ~60k characters of contract text>Output: one JSON object with all 41 CUAD clause names as keys, each a list of verbatim quotes (empty list = clause absent). Contracts longer than ~60k characters should be chunked with overlap and the per-chunk outputs merged (union per clause).
Usage
Ollama (recommended for local use)
ollama create cuad-extractor -f Modelfile # FROM ./cuad-4b-s42-q4_K_M.ggufCall with format: "json", temperature: 0, and num_ctx sized to fit your chunk plus the output (silent prompt truncation is the classic failure mode). On a small fraction of dense chunks (~6% in our runs) greedy decoding enters a repetition loop and hits the output cap; retrying that chunk with repeat_penalty: 1.05 (still temperature 0, deterministic) recovered nearly all cases in our evaluation, escalating to 1.15/1.25 for the rare holdouts.
Transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("Ihteshamstar/qwen3-4b-cuad-extractor",
torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("Ihteshamstar/qwen3-4b-cuad-extractor")Repository layout: fp16 merged weights at the root (from_pretrained works directly), gguf/ holds the q4KM quantization measured above, lora/ holds the LoRA adapter (r=16) if you prefer to apply it to the base model yourself.
Training
- Base: unsloth/Qwen3-4B-Instruct-2507, QLoRA (4-bit base, LoRA r=16, alpha=32, all attention + MLP projections)
- 619 chunk-level examples from the 408 official training contracts (one example whose rendered length exceeded the 24,576-token training window was excluded and recorded)
- 2 epochs, lr 2e-4 linear with 6% warmup, effective batch 8, ~45 min on one RTX 5090
- Target: the gold JSON only (trainonresponses_only); empty lists for absent clauses are part of the target, which is what teaches calibrated "not present" behavior
Limitations
- Strict-coverage recall is the weak spot (0.58): when a clause is stated in several places, the model often quotes the main span but not every occurrence.
- Greedy decoding can loop on dense chunks (see the fallback note above).
- Trained on CUAD: English-language commercial contracts from SEC EDGAR filings. Performance on other contract styles, other languages, or non-contract legal text is unmeasured.
- This is a research/engineering artifact, not legal advice; extractions need human review in any real workflow.