RUDXLABS/india-central-state-acts

India Central and State Acts Corpus

A scraped corpus of Indian central and state legislation from indiacode.nic.in, the Government of India's official legal code portal.

The PDFs are not in this repository. This repo holds the metadata index, the scraper source, and this documentation. The 34,729 PDFs (≈48 GB) are hosted on the Internet Archive at `india-central-state-acts`, packed into 256 zip shards. See Obtaining the PDFs.

Contents

• —34,729 unique PDFs of central and state acts (≈48 GB), deduplicated by SHA-256 content hash and stored under pdfs/{sha256}.pdf.
• —34,524 metadata documents in JSONL describing the source URL, title, per-document metadata, and references to the PDF files. Split across three collections that reflect the original scraping batches.
• —The source code (source/main.py) and the URL inventory (source/res-all-final.json) used to build the corpus.

License and source

These are public-domain government legal texts under §52(1)(q) of the Indian Copyright Act. They were obtained from the official Government of India portal, https://www.indiacode.nic.in/, in mid-2024.

Directory layout

.
├── README.md                  this file
├── MANIFEST.json              machine-readable summary and counts
├── pdfs/                      34,729 PDFs, named by SHA-256 content hash
│   └── {sha256}.pdf
├── metadata/                  Firestore-style JSONL index
│   ├── acts.jsonl             22,374 documents (largest batch)
│   ├── acts_new.jsonl         11,292 documents
│   └── central_acts.jsonl     858 documents
├── source/                    code used to assemble the corpus
│   ├── res-all-final.json     scraper output: all URLs + per-document metadata
│   ├── main.py                downloader that fetched each PDF into GCS
│   ├── requirements.txt
│   └── Dockerfile
└── extras/
    └── twing-test/            OCR test artifacts (Hindi/Malayalam)

Metadata schema

Each line in metadata/*.jsonl is one JSON document of the form:

{
  "id": "0004aaa2-268e-4670-8d52-0433165c2e1f",
  "source_url": "https://www.indiacode.nic.in/ViewSelectedActDetailsServlet?act_name=...",
  "metadata": { ... },
  "pdf_url": "https://...",
  "gcs_link": "pdfs/00038bbb...8e8.pdf",
  "other_links": [
    {
      "section": "...",
      "title": "Amendment...",
      "pdf_link": "https://...",
      "gcs_link": "pdfs/c7f9e7b2...673b4.pdf"
    }
  ]
}

gcs_link and the per-element other_links[].gcs_link are relative paths into pdfs/. The hash in the filename is the SHA-256 of the PDF bytes — you can verify integrity with shasum -a 256 pdfs/<file>.pdf.

Coverage caveats

• —Of the 34,524 metadata documents, roughly two thirds have a downloaded PDF; the remainder have an empty gcs_link because the upstream download failed (timeout, 404, redirect, malformed URL) at scrape time. The source URL is still preserved for everyone to re-fetch from indiacode.nic.in.
• —1,378 PDFs in pdfs/ are not referenced by any metadata document. They were uploaded to the storage buckets but the metadata write didn't complete. They are valid PDFs and can be inspected directly.
• —Parsing quality varies. Most PDFs are clean digitised text; a small fraction are scanned-image PDFs that produce noisier OCR.

Obtaining the PDFs

The PDFs live on the Internet Archive, sharded by the first two hex characters of each file's SHA-256 name — pdfs-00.zip through pdfs-ff.zip, 256 archives averaging ~190 MB. The shard for a given hash is therefore its first two characters, which makes the mapping computable without an index:

IA = "https://archive.org/download/india-central-state-acts"

def shard_url(sha: str) -> str:
    """URL of the zip shard containing a given PDF hash."""
    return f"{IA}/pdfs-{sha[:2]}.zip"

def pdf_url(sha: str) -> str:
    """Direct URL of a single PDF inside its shard."""
    return f"{IA}/pdfs-{sha[:2]}.zip/{sha}.pdf"

gcs_link fields in the metadata are of the form pdfs/{sha}.pdf, so strip the prefix and suffix to get the hash. To fetch one act's PDF:

import json, urllib.request

doc = json.loads(open("metadata/acts.jsonl").readline())
sha = doc["gcs_link"].removeprefix("pdfs/").removesuffix(".pdf")
urllib.request.urlretrieve(pdf_url(sha), f"{sha}.pdf")

To pull the whole corpus, download the shards rather than individual files:

pip install internetarchive
ia download india-central-state-acts --glob="pdfs-*.zip"

Every filename is the SHA-256 of the file's bytes, so integrity is verifiable offline with shasum -a 256 <file>.pdf.

How to use

As a flat PDF collection

Unpacking all 256 shards into one directory reproduces the original content-addressed store: look up a PDF by the hash in any metadata document's gcs_link field.

With Python

import json
docs = [json.loads(l) for l in open("metadata/acts.jsonl")]
print(len(docs), "documents")

# read the first document and open its PDF
d = docs[0]
print(d["source_url"])
print("PDF:", d.get("gcs_link", "(none)"))

With MongoDB (mongoimport)

for c in acts acts_new central_acts; do
  mongoimport --db legaling --collection $c --type json --file metadata/$c.jsonl
done

Then query as usual:

db.acts.find({ source_url: /GST|tax/i }).limit(5)

Rebuilding the corpus

source/res-all-final.json is a dict keyed by source URL, mapping each to its metadata. source/main.py is the downloader that takes that JSON and fetches each PDF. Running it requires Google Cloud credentials (a service account JSON that was originally bundled with the source has been deliberately removed for public release).

Stats

Acknowledgements and re-use

If you build something on top of this, an attribution is appreciated but not required. The underlying content belongs to the Government of India.

The corpus was originally assembled in 2024 as part of an internal AI experiment, abandoned, and re-published as a public dataset in 2026 after recovery. There is no ongoing maintenance — for fresh acts, scrape indiacode.nic.in directly.