sahilfarib/Legal-Document-Intelligence

<div align="center"> <h1>⚖️ Legal Document Intelligence System</h1> <p><b>A Hybrid Retrieval Pipeline for Grounded Legal Analysis</b></p>

     </div>

<br>

What this system does: This system converts messy legal documents into structured, grounded summaries using hybrid retrieval and citation-verified LLM drafting, with a feedback loop that adapts outputs based on operator edits.

Built as an advanced RAG (Retrieval-Augmented Generation) system tailored for the legal domain, prioritizing hallucination prevention and explicit traceability.

✨ Key Features

1. 1.Multi-Modal Ingestion & Structuring
2. 2.Routes PDFs and images through pdfplumber, Tesseract OCR, or a Gemini Vision fallback based on confidence heuristics.
3. 3.Extracts strongly-typed Pydantic fields (Parties, Dates, Obligations, Red Flags).

1. 1.Advanced Hybrid Retrieval
2. 2.Combines semantic vector search (ChromaDB + all-MiniLM-L6-v2) with sparse keyword search (rank_bm25).
3. 3.Scoring Logic: Retrieves the top 20 documents from both stores. Uses Reciprocal Rank Fusion formula: score = 1 / (k + rank) (where k=60) to balance exact match frequencies against semantic similarity.
4. 4.Reranker: The top 20 fused results are passed to a Cross-Encoder Reranker (ms-marco-MiniLM-L-6-v2) which outputs the final Top-6 Evidence Chunks.

1. 1.Post-Generation Citation Validation
2. 2.Gemini generates drafts with strict [CHUNK-id] citations.
3. 3.A structural sentence-by-sentence validator catches unsupported claims and flags uncited sentences.
4. 4.Validation Proof:
5. 5.Raw LLM Output: "The contract expires in 30 days. The vendor must pay $500 in late fees [CHUNK-a1b2c3d4]."
6. 6.Validator Flag: 🔴 Uncited sentence detected: "The contract expires in 30 days." (Calculates an uncited_ratio and warns the operator).

1. 1.Preference Adaptation (Feedback Loop)
2. 2.Captures operator edits via the Streamlit UI.
3. 3.Computes diffs, distills generalized writing rules, and saves them as structured JSON/text in SQLite.
4. 4.No model weight updates are performed. Rules are dynamically applied at the system prompt level only.
5. 5.Deduplicated via embedding similarity threshold to prevent prompt bloat.

📊 Evaluation Metrics

Measured on a synthetic dataset of mixed legal contracts and briefs (eval/ directory). Metrics were computed over 30 synthetic queries across 10 legal documents with manually annotated relevant chunks.

🚀 Quick Start

1. Prerequisites

• —Python 3.11+
• —Gemini API Key (Free Tier works great)
• —Optional: Docker

2. Local Installation

git clone https://github.com/sahilaf/Legal-Document-Intelligence-System.git
cd Legal-Document-Intelligence-System

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies deterministically
pip install -r requirements.txt

# Configure environment
cp .env.example .env
# Edit .env and add your GEMINI_API_KEY

3. Docker Installation (Recommended)

docker-compose up --build

App is available at `http://localhost:8501`

4. Run the Application (Local)

Start the Streamlit UI:

streamlit run ui/app.py

Navigate through the 4 tabs: Upload, Generate Draft, Review & Edit, and Learned Patterns.

Start the FastAPI server:

uvicorn api.main:app --reload --port 8000

Interactive API docs available at `http://localhost:8000/docs`.

📸 Visual Demo

(Add screenshots to `docs/assets/` to display here)

1. 1.Upload Screen: [Screenshot Placeholder: upload.png]
2. 2.Retrieval Evidence Panel: [Screenshot Placeholder: retrieval.png]
3. 3.Generated Draft with Citations: [Screenshot Placeholder: draft.png]
4. 4.Learned Pattern List: [Screenshot Placeholder: patterns.png]

🔄 Example Workflow

1. 1.Input Document: Upload a noisy, scanned NDA (data/sample_inputs/).
2. 2.Structuring: System extracts structured fields (e.g., Parties: Company A, Company B).
3. 3.Retrieval: User queries "Summarize termination obligations." System retrieves CHUNK-12, CHUNK-44.
4. 4.Draft Generation: System generates text: "The contract terminates in 30 days [1]."
5. 5.Validation: Validator confirms [1] points to CHUNK-12 and calculates uncited_ratio=0.03.
6. 6.Operator Edit: Operator changes "30 days" to "thirty (30) days".
7. 7.Learned Pattern: System distills: "Always spell out numbers and include digits in parentheses" and saves to SQLite.

🧠 System Architecture

[image]

Design Decisions:

• —BM25 + Embeddings: BM25 handles exact legal terminology, while embeddings capture semantic similarity.
• —RRF + Reranker: RRF balances sparse+dense retrieval, and the reranker improves precision over simple cosine similarity.
• —Structural Validator: Explicitly reduces hallucination risk without relying solely on LLM self-correction.
• —Prompt-Level Adaptation: "Learning" is achieved via preference extraction and injection, avoiding the overhead of continuous fine-tuning.

⚠️ Limitations & Real-World Constraints

• —Handwritten Cursive: Still difficult for the OCR fallback pipeline.
• —Low DPI Scans: OCR degrades heavily on low DPI or warped images.
• —Validator Scope: The validator is a structural check (ensuring citations exist and map to real chunks), not semantic truth verification.
• —Context Window: Fixed-size chunking (512 chars) can occasionally lose long-range context in complex legal arguments.
• —Rate Limits: Gemini free tier quota exhaustion is caught gracefully, triggering explicit UI warnings.
• —Malformed PDFs: Handled via empty retrieval state fallbacks.

⏱️ Latency & Cost Breakdown

• —End-to-End Latency: Averages 4.2s per query.
• —Retrieval: ~0.1s
• —Reranking: ~0.8s (Compute overhead of the local cross-encoder model)
• —LLM Drafting: ~3.1s
• —Validation: ~0.2s
• —API Cost: Zero. The system uses the Gemini 2.5 Flash Free Tier (which permits 15 RPM). Vector embeddings and reranking run 100% locally.

🔌 API Reference

1. Ingestion

curl -X POST http://localhost:8000/ingest -F "file=@data/sample_inputs/contract_messy.txt"

2. Draft Generation

curl -X POST http://localhost:8000/draft \
  -H "Content-Type: application/json" \
  -d '{"query": "Summarize termination obligations", "draft_type": "Internal Memo"}'

3. Pattern Management

curl -X GET http://localhost:8000/patterns

🔐 Security & Observability

• —Local Inference: Embeddings and reranking run 100% locally.
• —No Document Persistence: Documents are not persisted outside of the local storage/ volume.
• —API Key Isolation: API keys are injected at runtime via environment variables.
• —Database: SQLite is used locally.
• —Observability: Logging is handled via loguru with async background processing in FastAPI for non-blocking execution.

📂 Project Structure

├── api/                # FastAPI endpoints and dependencies
├── data/               # Sample legal documents for testing
├── eval/               # Evaluation scripts (Precision@K, Feedback metrics)
├── feedback/           # Diff engine, pattern distillation, and SQLite store
├── generation/         # LLM prompting, drafting, and citation validation
├── ingestion/          # OCR, chunking, and structured extraction
├── retrieval/          # ChromaDB, BM25, Hybrid RRF, and Reranking
├── storage/            # Local databases (ChromaDB, SQLite, BM25 pickles)
├── tests/              # Comprehensive pytest suite (59/59 passing)
├── ui/                 # Streamlit interface

🔮 Future Improvements

• —Layout-Aware Parsing: Improve table and multi-column extraction.
• —Legal Ontology Integration: Enhance retrieval with predefined legal concepts.
• —Active Learning: Allow the system to ask clarifying questions when operator edits are ambiguous.
• —Caching Layer: Cache embeddings and OCR outputs to mitigate API quotas.

📜 License

Distributed under the MIT License. See LICENSE for more information.