tojpaj/legal-aid-routing-model

NGO Legal Aid — Intake Routing Model

Built for the [Adaption AutoScientist Challenge](https://adaptionlabs.ai/blog/autoscientist-challenge) — Legal Category, Round 1 🏆 This project would not exist without [Adaption](https://adaptionlabs.ai) and its AutoScientist platform. The quality leap that made this model work — a D-to-A jump in training-data quality — was done entirely by AutoScientist, automatically. See below. ⚡ The multilingual Gemma 4 LoRA adapter, the RAG grounded-answer layer, and the voice-input transcription in this project are all powered by [Together AI](https://www.together.ai) — making large-model inference accessible, affordable, and fast.

🎯 Intended use & context

The problem: the intake bottleneck kills access to justice

Every legal aid NGO faces the same crisis: far more people need help than there are specialists to serve them. The triage step — reading each intake submission and deciding which practice area it belongs to (housing? family? domestic violence? immigration?) — is done manually, case by case, by a caseworker who already has a full load.

This creates a queue. Cases sit in a generic backlog waiting to be read and routed. For most matters, a delay of a day or two is an inconvenience. For a subset of cases, it is catastrophic:

• —A domestic violence survivor waiting for a restraining order cannot go home.
• —A family facing eviction has days, not weeks, before they are on the street.
• —An immigration detainee has a hearing date that will not move.
• —A parent in a child custody dispute may lose visitation while their case waits.

The routing decision itself is not a specialist task — it takes a trained caseworker about 30 seconds to read a description and assign a category. But those seconds add up across hundreds of daily intakes. The bottleneck is not legal reasoning; it is triage volume.

What this system does

This model reads the client's own words — exactly as they would write them on an intake form or speak them at a walk-in clinic — and instantly predicts which of 16 practice areas the case belongs to, so it can be routed to the right specialist queue without a human reading it first.

"My landlord hasn't fixed the heating and now says I owe back rent."
                          ↓  model  ↓
              housing  (82%)  •  courts (7%)  •  consumer (4%)

That 82%-confidence routing fires in under 50 ms on CPU. The specialist for housing law sees the case before the intake worker even closes the browser tab.

Who benefits, and how

Languages & accessibility

The system is designed for the linguistic and literacy reality of who legal aid actually serves:

• —English — default, full-accuracy (74.2% accuracy / 60.0% macro-F1)
• —Hindi — Gemma 4 LoRA adapter + RAG retrieval (multilingual bge-m3 embedder)
• —Punjabi (Gurmukhi) — same Gemma 4 adapter + RAG
• —Voice input — Whisper Large v3 (Together AI), 50+ languages, $0.0015/audio minute; designed for low-literacy users, clients with disabilities, and elderly users who cannot type

The fast-track resolution angle

The AutoScientist Challenge asked for solutions to "fast-track resolution" in the legal domain. This system addresses the resolution bottleneck at its earliest, highest-leverage point: the moment a client first describes their problem. Every second saved in triage is a second added to the specialist's time with the client. For time-sensitive cases, routing in seconds versus hours is not a marginal improvement — it is the difference between getting help in time and not.

Out-of-scope / not intended for

• —Legal advice — this is a routing and retrieval tool. It identifies categories and cites statutes. Always refer clients to a qualified lawyer for substantive advice.
• —Autonomous case decisions — routing suggestions must be reviewed by a caseworker. Do not use model output as the sole basis for accepting, rejecting, or prioritising a case.
• —Other jurisdictions — the classifier was trained on US LegalBench/LearnedHands data; the RAG layer covers Indian criminal law (BNS/BNSS/BSA). Neither is validated for other legal systems.
• —Civil statutes in India — the RAG knowledge base covers criminal codes. A production system would additionally index tenancy, labour, family, and consumer statutes.

🌟 Why Adaption AutoScientist made this possible

Legal-aid intake data is inherently messy. Real posts from people in crisis are short, emotional, vague, full of abbreviations, missing context — the kind of text that confuses classifiers trained on clean benchmarks. Before AutoScientist, the raw LegalBench data had a quality score of 4.0 / 10 (Grade D, 7.6th percentile) — low-quality prompts, near-zero completion quality (avg 1.02 / 10).

AutoScientist — Adaption's automated AI-scientist platform — transformed that dataset without any manual prompt engineering:

The enhanced_prompt column AutoScientist produced became the actual training signal for this model. The +11.2% macro-F1 improvement over the TF-IDF baseline — which matters enormously for rare practice areas like education, benefits, and immigration where clients most need help — is a direct consequence of that data quality lift.

Without AutoScientist, we'd be training on Grade-D data. The model would be training on noise. AutoScientist is what turned this from a low-quality experiment into a submission-ready system.

What AutoScientist does (briefly)

Adaption built AutoScientist to act as an automated research scientist for AI data: it analyses your dataset, hypothesises improvements, runs controlled experiments, and selects the transformations that verifiably improve training quality — all without you writing a single data-cleaning script. You bring a CSV; it brings a Grade-A dataset.

• —SDK: docs.adaptionlabs.ai
• —Challenge: adaptionlabs.ai/blog/autoscientist-challenge

⚡ Together AI — powering the multilingual & voice layers

Together AI provides the infrastructure that makes the large-model components of this project viable at low cost:

The DistilBERT classifier and FAISS retrieval run locally for free. Together AI is invoked only for tasks requiring large multilingual models — keeping inference costs predictable.

• —Together AI: together.ai
• —Together AI docs: docs.together.ai
• —Models used: google/gemma-4-31B-it, openai/whisper-large-v3

🗂️ What's in this repo

⚡ Quick start — classifier

from transformers import pipeline

clf = pipeline(
    "text-classification",
    model="tojpaj/legal-aid-routing-model",
    top_k=5,
)

# English example
result = clf("My landlord is trying to evict me without giving proper notice.")
for r in result[0]:
    print(f"{r['label']:22s}  {r['score']:.1%}")
# housing                 82.3%
# courts                   6.9%
# family                   4.1%
# ...

16 practice areas: benefits · business · consumer · courts · crime · divorce · domestic_violence · education · employment · estates · family · health · housing · immigration · torts · traffic

🔍 Quick start — RAG (Indian statute retrieval)

Retrieve relevant sections from the Bharatiya Nyaya Sanhita / Nagarik Suraksha Sanhita / Sakshya Adhiniyam 2023 in English, Hindi, or Punjabi. Embedding and retrieval run locally for free; grounded summaries use the Together AI API.

Installation

pip install faiss-cpu sentence-transformers pandas pyarrow
# Optional — for Gemma 4 grounded summaries:
pip install together

Pure retrieval (no API key needed)

import sys
sys.path.insert(0, "rag/")   # path to rag.py + rag_index.faiss + rag_meta.parquet
import rag

# English
sections = rag.retrieve("My employer has not paid my wages for three months.", k=5)
for s in sections:
    print(f"[{s['act']} §{s['section_number']}] {s['section_title']}  (score: {s['score']:.3f})")
    print(s["text"][:300], "\n")

# Hindi
sections = rag.retrieve("मेरे मकान मालिक ने बिना नोटिस मुझे निकाला।", k=3)

# Punjabi (Gurmukhi)
sections = rag.retrieve("ਮੇਰੇ ਮਾਲਕ ਨੇ ਮੈਨੂੰ ਬਿਨਾਂ ਨੋਟਿਸ ਕੱਢ ਦਿੱਤਾ।", k=3)

Grounded answer (Gemma 4 via Together AI)

import os, sys
os.environ["TOGETHER_API_KEY"] = "your_together_api_key"
sys.path.insert(0, "rag/")
import rag

answer, sources = rag.grounded_answer(
    "My employer hasn't paid me for 3 months. What does Indian law say?",
    k=5,
)
print(answer)
print("\n--- Cited sections ---")
for s in sources:
    print(f"  [{s['act']} §{s['section_number']}] {s['section_title']}")

Example output:

Under the Bharatiya Nyaya Sanhita 2023, withholding earned wages may
constitute criminal breach of trust (BNS §316) or cheating (BNS §318)
depending on circumstances. Additionally, BNSS §223 outlines how to
approach a magistrate to file a cognizance complaint...

--- Cited sections ---
  [BNS §316] Criminal breach of trust
  [BNS §318] Cheating
  [BNSS §223] Cognizance of offences by magistrates
  [BNS §3]   Definitions
  [BNSS §35]  Arrest without warrant

📊 Results

Evaluated on the held-out 755-row test.csv. Macro-F1 is the key metric — accuracy is dominated by the majority "housing" class. The +11.2% macro-F1 gain means meaningfully better routing for rare practice areas where clients most need correct routing (education, benefits, immigration, domestic violence).

🌐 Multilingual extension (English / Hindi / Punjabi)

A LoRA adapter for google/gemma-4-31B-it (fine-tuned via Together AI) extends routing to Hindi and Punjabi with a short preliminary analysis. Training data: 2,898 rows = 1,000 EN + 1,000 HI + 898 PA — all sourced from the AutoScientist `enhanced_prompt`, not the raw pre-adaptation text.

MuRIL cross-lingual transfer experiment (zero-shot Hindi/Punjabi):

Transfer is real — Hindi accuracy tracks within ~3 pts of English with zero Hindi training examples.

🇮🇳 Native Indian-law RAG (rag/)

The classifier is trained on US LegalBench content (honestly documented). The RAG layer patches this: retrieval only needs embeddable statute text — no intake labels — so the RAG layer is genuinely grounded on real Indian law, not translated US content.

Knowledge base: `GSMS-B/indian-legal-sections-bns-bnss-bsa-2023` (Apache-2.0)

• —BNS — Bharatiya Nyaya Sanhita 2023 (replaced Indian Penal Code 1860)
• —BNSS — Bharatiya Nagarik Suraksha Sanhita 2023 (replaced CrPC 1973)
• —BSA — Bharatiya Sakshya Adhiniyam 2023 (replaced Evidence Act 1872)

Embedder: BAAI/bge-m3 — multilingual (EN/HI/PA), runs locally, free.

Reproduce the FAISS index from scratch

pip install faiss-cpu sentence-transformers datasets pandas pyarrow
python rag/build_rag_kb.py
# → rag_index.faiss + rag_meta.parquet (takes ~5 min on CPU)

🎤 Full demo app (Gradio)

pip install gradio transformers torch joblib together sentence-transformers faiss-cpu
python rag/demo_app.py

Three tabs:

1. 1.Type — paste legal problem, get top-5 practice areas live
2. 2.Speak (accessibility) — voice input via Whisper (50+ languages), transcribed then routed
3. 3.Know your rights (Indian law · RAG) — ask in EN/HI/PA, get statute-grounded answer with act/section citations

🔁 Full reproducibility

git clone https://huggingface.co/tojpaj/legal-aid-routing-model
cd legal-aid-routing-model
pip install transformers datasets scikit-learn torch \
    sentence-transformers faiss-cpu gradio together

# 1. Rebuild the dataset from LegalBench
python scripts/build_dataset.py          # → train.csv, test.csv

# 2. Run the TF-IDF baseline
python scripts/baseline.py               # → baseline_model.joblib, baseline_results.json

# 3. Fine-tune DistilBERT (needs GPU, ~1 hr on T4)
python scripts/train_model.py            # → model.safetensors

# 4. Rebuild the Indian-law FAISS index
python rag/build_rag_kb.py               # → rag_index.faiss, rag_meta.parquet

# 5. Launch the Gradio demo
python rag/demo_app.py

📋 Links

⚠️ Limitations

• —Classifier trained on US legal-aid content (LegalBench/LearnedHands) translated to Hindi/Punjabi — not a native Indian-law classifier. The RAG layer grounds on real Indian statutes, but the classifier labels are US-sourced.
• —RAG covers India's criminal codes (BNS/BNSS/BSA); a production system would additionally index civil statutes (tenancy, labour, family, consumer).
• —Preliminary information only — not legal advice.