soybelli/gemma-4-E4B-it-cuad-lora

Gemma 4 E4B — CUAD Contract Review (LoRA)

A LoRA adapter that fine-tunes Gemma 4 E4B (instruct) for legal contract clause review on the CUAD (Contract Understanding Atticus Dataset).

Given a contract excerpt and a clause question (e.g. "Highlight the parts related to 'Governing Law'…"), the model extracts the exact relevant span from the excerpt, or answers "Not found" when the clause is absent.

• —Base model: unsloth/gemma-4-E4B-it (trained on the unsloth/gemma-4-E4B-it-unsloth-bnb-4bit 4-bit variant)
• —Method: QLoRA SFT with Unsloth + TRL (r=16, alpha=16), language layers only
• —Task: extractive QA / clause highlighting over 41 CUAD clause categories
• —Adapter size: ~37M trainable params (0.46% of the base)

Results

Evaluated on held-out contracts (10% of CUAD held out by contract, so no context leakage), SQuAD-2.0-style scoring on 80 questions (40 answerable / 40 not-found):

The base model already abstains well on absent clauses; the fine-tune's main gain is returning exact clause spans instead of verbose paraphrases, with no regression on the not-found class.

Usage

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel

BASE = "unsloth/gemma-4-E4B-it"
ADAPTER = "soybelli/gemma-4-E4B-it-cuad-lora"

processor = AutoProcessor.from_pretrained(BASE)
model = AutoModelForImageTextToText.from_pretrained(BASE, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER).eval()

SYSTEM = (
    "You are a legal contract review assistant. Read the contract excerpt and answer the question. "
    "If the relevant clause is present, quote the exact text from the excerpt. "
    "If it is not present, reply exactly: Not found."
)
excerpt = "...THIS AGREEMENT shall be governed by and construed in accordance with the laws of the State of New York..."
question = (
    'Highlight the parts (if any) of this contract related to "Governing Law" that should be reviewed by a '
    "lawyer. Details: Which state/country's law governs the interpretation of the contract?"
)
prompt = f"{SYSTEM}\n\n### Contract excerpt:\n{excerpt}\n\n### Question:\n{question}\n\n### Answer:"

messages = [{"role": "user", "content": [{"type": "text", "text": prompt}]}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True, return_tensors="pt", return_dict=True
).to(model.device)
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip())

Training

• —Data: CUAD SQuAD-2.0 JSON (CUAD_v1/CUAD_v1.json), flattened to (excerpt, question, answer) pairs. Excerpts are ~2,400-char windows centered on the answer span; the natural ~32% answerable / 68% not-found distribution is kept. 18,759 training examples (10% of contracts held out for eval).
• —Config: 1 epoch, max_seq_length=1024, effective batch 16, lr 2e-4, adamw_8bit, bf16, trained only on the assistant response. Final train loss ≈ 0.14.
• —Hardware: single NVIDIA RTX 5090 (~95 min).

Limitations

• —Trained on ~2,400-char (≈600-token) excerpts; for full contracts, chunk the document and query each chunk.
• —English commercial contracts only; not legal advice.
• —Inherits the base model's and CUAD's biases. CUAD is licensed CC-BY-4.0.