epequeno/legal-entailment-deberta-v3-large

legal-entailment-deberta-v3-large (v2)

A cross-encoder/nli-deberta-v3-large model fine-tuned for legal citation verification — determining whether a cited contract passage actually supports a generated claim. Built for LegalKit, an open-source legal AI platform.

Performance: Honest Numbers

We report two evaluations. The expanded evaluation is the more reliable and representative metric.

Expanded Evaluation (243 cases, 14 categories)

Includes 8 in-distribution categories the model was trained on, plus 6 out-of-distribution categories (representations & warranties, limitation of liability, force majeure, assignment/change of control, governing law, non-compete) it has never seen.

Per-label accuracy:

In-Distribution Test Split (453 examples)

This is the standard held-out test split from the training data. It overstates real-world performance because it shares the same data generation methodology as training.

Comparison with Base Model

The fine-tuning provides a genuine ~9pp improvement over the base DeBERTa model on legal text. The model is most valuable for its 89.1% accuracy on contradictions — the critical class for catching incorrect citations.

Known Issues

1. The partially_entailed class is unreliable

An audit of 100 training labels revealed that 84% of `partially_entailed` labels in the training data are wrong — most should be entailed. The model's 15.3% accuracy on this class reflects noisy training data, not a learning failure.

Impact: The 4-class accuracy (50.6%) is dragged down by this broken class. For production use, we recommend collapsing to binary: entailed + partially_entailed → SUPPORTED, contradicted + neutral → NOT SUPPORTED.

2. Training data quality varies by source

A v3 model is planned that will: (a) collapse to 3 classes, (b) use corrected labels, (c) be evaluated on the expanded 243-case benchmark.

3. Original 33-case eval was too small

Our initial model card reported 87.9% binary accuracy on 33 hand-labeled cases. This was within the 8 trained-on categories only. The expanded 243-case eval (including OOD categories) gives the more realistic 72.8% figure.

Model Description

A cross-encoder that takes a (passage, claim) pair and classifies the relationship:

Primary use case: Citation verification in RAG pipelines over legal contracts. After an LLM generates an answer with citations, this model checks whether each cited passage supports its associated claim.

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tokenizer = AutoTokenizer.from_pretrained("epequeno/legal-entailment-deberta-v3-large")
model = AutoModelForSequenceClassification.from_pretrained("epequeno/legal-entailment-deberta-v3-large")
model.eval()

labels = ["entailed", "partially_entailed", "contradicted", "neutral"]

passage = 'The aggregate liability of Seller under this Article VIII shall not exceed Five Million Dollars ($5,000,000).'
claim = "The indemnification cap is $5,000,000."

inputs = tokenizer(passage, claim, return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
    logits = model(**inputs).logits
    pred = logits.argmax(-1).item()

print(f"Prediction: {labels[pred]}")  # → entailed

# For production binary use:
supported = pred in [0, 1]  # entailed or partially_entailed
print(f"Supported: {supported}")

Training Details

Training Data Sources

Strengths

• —89.1% accuracy on contradictions — the most critical class for citation verification
• —Zero marginal cost — runs locally, no API calls needed
• —139ms latency — fast enough for real-time checking
• —Genuine improvement over base DeBERTa on legal-specific patterns (notwithstanding clauses, shall/may semantics, MAE carve-outs)

Limitations

• —partially_entailed class is unreliable (see known issues)
• —Optimized for US commercial contracts — jurisdiction transfer not validated
• —Temporal arithmetic remains challenging ("18 months" vs "2 years")
• —Max sequence length 512 tokens
• —53.8% of training data has unverified LLM-generated labels

Roadmap

• —v3 (planned): Collapse to 3 classes, corrected labels, expanded eval benchmark, expected binary accuracy improvement to ~80%+

Related Resources

• —LegalKit — open-source legal AI platform
• —legal-embeddings-bge-base — companion retrieval model
• —legal-entailment-benchmark — evaluation dataset
• —Base model: cross-encoder/nli-deberta-v3-large

Citation

@software{legalkit2026,
  author = {Pequeno, Steven},
  title = {LegalKit: Open-Source Legal AI Platform},
  year = {2026},
  url = {https://github.com/legalkit/legalkit}
}