datgacon/cuad-cross-encoder-v11

cuad-cross-encoder-v11

A cross-encoder reranker fine-tuned for legal clause retrieval in contract review workflows. Built on `cross-encoder/ms-marco-MiniLM-L-6-v2` and fine-tuned on a combination of CUAD, ACCORD, LEDGAR, ContractNLI, EDGAR-sourced contract pairs, and LLM-generated synthetic pairs targeting known failure patterns.

Deployed as ONNX INT8 for in-browser inference via ONNX Runtime Web.

Intended Use

• —Primary: Reranking retrieved contract chunks against natural-language clause queries (e.g. "What are the governing law provisions?", "What IP does each party retain?")
• —Domains covered: Joint Venture, Intellectual Property, Non-Compete / Non-Solicit, Non-Disclosure Agreement (NDA)
• —Not intended for: General-purpose document retrieval, non-legal domains, or as a standalone legal advisor

Training Data

Total: ~72,101 training pairs · 5,612 validation pairs

Pairs are (query, positive_chunk, negative_chunk) triplets. Negatives are a mix of hard negatives (wrong clause from same contract) and random negatives (chunks from other contracts).

Training Details

Evaluation

Evaluated on a held-out set of 16 contracts across 4 clause domains. Each contract is queried with 3–8 clause-type questions; the top-ranked chunk is scored as pass (correct clause returned), partial (correct section but wrong chunk boundary), or fail.

Test contracts (JV): MightyCell Batteries, BorrowMoney.com, Galera Therapeutics, MINDA IMPCO Technologies, Kiromic Biopharma, Novo Integrated Sciences, Transphorm / Aizu Fujitsu, Valence Technology / Baoding Fengfan, Veoneer

Test contracts (IP): Armstrong Flooring, Cerence Inc, Garrett Motion, Rare Element Resources

Test contracts (NDA): Kite Pharma / Gilead Sciences, Fortune Brands / Norcraft Companies, Aspect Medical Systems / Tyco Healthcare

The JV improvement (+11 pass) is driven by new Sino-JV EDGAR data and synthetic definitions-bleed pairs targeting contracts where the model previously returned definitions articles for Governing Law and Non-Compete queries.

Usage

ONNX Runtime (recommended for browser / edge)

import onnxruntime as ort
from transformers import AutoTokenizer
import numpy as np

tokenizer = AutoTokenizer.from_pretrained("datgacon/cuad-cross-encoder-v11")
session = ort.InferenceSession("onnx/model_quantized.onnx")

query = "What governing law applies to this agreement?"
passage = "This Agreement shall be governed by and construed in accordance with the laws of the State of Delaware."

inputs = tokenizer(query, passage, return_tensors="np", max_length=512, truncation=True, padding=True)
outputs = session.run(None, {k: v for k, v in inputs.items() if k in ["input_ids", "attention_mask", "token_type_ids"]})
score = outputs[0][0][0]
print(f"Relevance score: {score:.4f}")

sentence-transformers (PyTorch)

from sentence_transformers.cross_encoder import CrossEncoder

model = CrossEncoder("datgacon/cuad-cross-encoder-v11")

query = "What governing law applies to this agreement?"
passages = [
    "This Agreement shall be governed by the laws of the State of Delaware.",
    "Each party shall maintain the confidentiality of the other party's information.",
    "The term of this Agreement shall commence on the Effective Date.",
]

scores = model.predict([(query, p) for p in passages])
ranked = sorted(zip(scores, passages), reverse=True)
for score, passage in ranked:
    print(f"{score:.4f}  {passage[:80]}")

Limitations

• —Trained on US commercial contracts (CUAD corpus); may underperform on EU, UK, or public-sector agreements
• —Partial matches are common at clause-boundary edges — chunk size and overlap in the retrieval pipeline significantly affect results
• —Not a legal advisor — scores indicate retrieval relevance, not legal interpretation
• —Performance on clause types outside the four trained domains (JV, IP, NC, NDA) is untested
• —Token type IDs must be passed explicitly when using ONNX Runtime Web; omitting them collapses score spread

Citation

If you use this model, please cite the underlying datasets:

@article{hendrycks2021cuad,
  title={CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review},
  author={Hendrycks, Dan and others},
  journal={arXiv preprint arXiv:2103.06268},
  year={2021}
}