antonhome/indian-legal-supervised-fine-tuning-data

🇮🇳 LegalBrain Indic Legal Corpus

A large-scale multilingual Indian legal dataset curated to support research in:

• —Domain-specific LLM training
• —Legal question answering
• —Policy reasoning & case retrieval
• —Agentic systems for legal workflow automation

This dataset contains text drawn from publicly available legal sources across multiple Indian languages, including: English, Hindi, Marathi, Bengali, Kannada, Tamil, Telugu, Odia, and others.

The corpus is structured and processed to be directly usable for supervised fine-tuning, RAG pipelines, and conversational legal assistants.

📦 Dataset Structure

After preprocessing and supervised alignment, the dataset is provided in the format:

This enables training of:

• —Legal chatbots
• —Agentic reasoning systems
• —Legal retrieval-augmented QA models
• —Court case summarizers
• —Argumentation-based LLM pipelines

🏛️ Data Sources

Data was collected only from publicly and legally accessible sources, including:

• —Supreme Court judgments
• —High Court decisions
• —Law Commission reports
• —Public legal textbooks & commentaries
• —Open legal news archives
• —Public domain legal Q&A portals
• —Government acts, rules, and notifications

No proprietary or licensed content was used.

🧹 Cleaning & Normalization Pipeline

Large-scale legal data is noisy. The following steps were used:

1. 1.HTML + Boilerplate Removal Removal of menus, footers, ads, repeated headers, legal boilerplate markers.

1. 1.OCR + Text Correction OCR applied to scanned PDFs using Tesseract + custom normalization, followed by regex-based cleanup for:
2. 2.Section markers
3. 3.Citations
4. 4.Case line references

1. 1.Language Detection & Segmentation Auto-sharding by language → Sentence & clause-level segmentation using spaCy + Indic NLP.

1. 1.De-duplication Removed near-duplicate clauses across multiple case reports using MinHash (LSH).

✨ Argilla-Based Supervised Dataset Construction

To transform unstructured text into (context, question, response) triplets, the dataset was processed using Argilla for human feedback and model-assisted annotation.

Workflow:

1. 1.Select meaningful legal text chunks (150–600 words).
2. 2.Use a prompting pipeline to generate candidate questions and answers.
3. 3.Load candidate examples into Argilla workspace.
4. 4.Curate + refine:
5. 5.Fix hallucinations
6. 6.Improve citation grounding
7. 7.Ensure all responses strictly reference context
8. 8.Export final reviewed dataset to HF.

This ensures the dataset trains models that cite the law instead of hallucinating it.

🏗️ Example Entry

{
  "context": "The right to constitutional remedies allows citizens to approach the Supreme Court under Article 32...",
  "question": "Which constitutional provision allows an individual to directly move the Supreme Court?",
  "response": "Article 32 of the Constitution grants the right to constitutional remedies, enabling citizens to directly approach the Supreme Court."
}