Modelpublic

cs-552-2026-Clanker-Scientists/safeguard-deberta-contractnli-v2

sourceHugging Facemitupdated 4mo agoView on Hugging Face

0likes11downloads

Model Card

Safeguard v2 — DeBERTa-v3-large-mnli FT on Stanford ContractNLI

Component of CS-552 Spring 2026 multi-agent legal QA pipeline (Clanker-Scientists team). Acts as a hallucination guard: given a contract excerpt + draft claim, decides whether the contract supports the claim.

Training

—Base: MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli
—Data: Stanford ContractNLI, 8,228 NDA premise/hypothesis/label triples
—Method: full fine-tuning, 3 epochs, LR 2e-5
—Key methodological choice: preserve the pretrained 3-class NLI head (entailment/neutral/contradiction). Binarize at inference via argmax == entailment_class. Re-initializing as a 2-class head (v1) lost 0.075 macro-F1.

Evaluation

Test set	Metric	Score
LegalBench `contract_nli_explicit_identification` N=100 seed=0	macro-F1	0.838
Phase-1 baseline (Qwen3-14B as judge)	macro-F1	0.831

Beats the LLM-judge baseline by +0.007 at ~600× lower inference cost.

Usage

python

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tok = AutoTokenizer.from_pretrained("cs-552-2026-Clanker-Scientists/safeguard-deberta-contractnli-v2")
model = AutoModelForSequenceClassification.from_pretrained("cs-552-2026-Clanker-Scientists/safeguard-deberta-contractnli-v2")
model.eval()

premise = "The Receiving Party shall not disclose any Confidential Information..."
hypothesis = "The Agreement explicitly identifies confidential information as such."

inputs = tok(premise, hypothesis, truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
pred = logits.argmax(-1).item()
# 0 = entailment (supported), 1 = neutral (unsupported), 2 = contradiction (unsupported)
supported = (pred == 0)
print("Supported" if supported else "Unsupported")