Kamalesh-genai/phi3-mini-legal-clause-lora

Phi-3-mini LoRA — Legal Contract Clause Classification

This is a LoRA (QLoRA) adapter fine-tuned on top of microsoft/Phi-3-mini-4k-instruct for legal contract clause classification, trained on a subset of the CUAD (Contract Understanding Atticus Dataset).

Model Details

• —Base model: microsoft/Phi-3-mini-4k-instruct (3.8B params)
• —Method: QLoRA (4-bit NF4 quantization) + PEFT LoRA (r=16, alpha=32)
• —Task: Multi-class contract clause classification (8 categories: Parties, License Grant, Cap On Liability, Anti-Assignment, Audit Rights, Insurance, Expiration Date, Governing Law)
• —Training framework: Hugging Face transformers, peft, trl (SFTTrainer)
• —Hardware: Google Colab free tier, 1x T4 GPU (16GB VRAM)
• —Training data: ~1,000 clause examples from CUAD, 2 epochs
• —Adapter size: ~17.8 MB

Evaluation

• —98.00% accuracy on a 50-example held-out eval slice (exact-match classification)
• —Final training loss: 0.81, validation loss: 0.77, mean token accuracy: ~82.9%

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

base_model_id = "microsoft/Phi-3-mini-4k-instruct"
adapter_id = "Kamalesh-genai/phi3-mini-legal-clause-lora"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id, quantization_config=bnb_config, device_map="auto"
)
model = PeftModel.from_pretrained(base_model, adapter_id)

prompt = (
    "<|user|>\nYou are a legal contract analyst. Classify the following "
    "contract clause into exactly ONE of these categories: Parties, "
    "License Grant, Cap On Liability, Anti-Assignment, Audit Rights, "
    "Insurance, Expiration Date, Governing Law.\n\n"
    "Clause:\n\"\"\"This Agreement shall be governed by the laws of the "
    "State of New York.\"\"\"\n\nCategory:<|end|>\n<|assistant|>\n"
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=10)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Training Details

• —Dataset: CUAD (Atticus Project), filtered to the 8 most frequent clause categories
• —LoRA config: r=16, alpha=32, dropout=0.05, target modules: q/k/v/oproj, gate/up/downproj
• —Optimizer: pagedadamw8bit, lr=2e-4, cosine schedule
• —Batch size: 2 (effective 8 with gradient accumulation), 2 epochs

Limitations

• —Trained on a filtered subset (top-8 CUAD categories, ~1,000 examples), not the full 41-category CUAD benchmark
• —Classification only — does not extract clause spans (original CUAD QA task)
• —Evaluated with simple exact-match accuracy on a small held-out slice

Author

Fine-tuned by Kamaleshwar S as a portfolio project — training code on GitHub