Datasetpublic

issai/transboundary-water-governance-corpus

Global Transboundary Water Governance Corpus A curated, openly available corpus of 252 official policy, legal, and institutional documents that govern internationally shared waters (rivers, lakes, and aquifers), released with full extracted text and harmonised structured metadata. Coverage: 123 river basins, lake systems, and hydrological regions; 141 sovereign states as parties; 14 languages; documents published 1909–2025. Content types: treaties, agreements, conventions… See the full description on the dataset page: https://huggingface.co/datasets/issai/transboundary-water-governance-corpus.

sourceHugging Facecc-by-4.0updated 1mo agoView on Hugging Face

0likes234downloads

Dataset Card

Global Transboundary Water Governance Corpus

A curated, openly available corpus of 252 official policy, legal, and institutional documents that govern internationally shared waters (rivers, lakes, and aquifers), released with full extracted text and harmonised structured metadata.

—Coverage: 123 river basins, lake systems, and hydrological regions; 141 sovereign states as parties; 14 languages; documents published 1909–2025.
—Content types: treaties, agreements, conventions, protocols, national laws, strategies, action plans, and framework documents issued by governments, intergovernmental organisations, basin authorities, and United Nations agencies.
—Size: ~59 million characters of clean UTF-8 text. 233 records hold the full document body; 19 are landing- or press-page text only (flagged via source.status).

This deposit accompanies a Data in Brief data article that describes it in full.

Repository structure

corpus/               252 UTF-8 text files, one per document, named <NN>-slug.txt
documents.json        252 metadata records (one JSON object per document)
corpus_summary.csv    flat one-row-per-document table of the core metadata
fetch_corpus.py       retrieval + text-extraction script that builds the corpus
README.md             this file (dataset card + data dictionary)

The <NN> prefix is a permanent document number, stable across releases and referenced by source.text_path in the metadata.

Metadata schema (`documents.json`)

Each record has descriptive fields plus a source provenance block.

Field	Description
`title`	Official document title
`type`	Document type (agreement, treaty, convention, protocol, law, strategy, plan, policy, framework, etc.)
`level`	Governance level: basin / national / regional / global
`issuer`	Issuing body (government, IGO, basin authority, UN agency, etc.)
`basin_or_region`	River basin, lake system, hydrological region, or "global" framework scope
`countries`	Array of parties / sovereign states involved
`year`	Release or implementation year
`language`	Original document language (notes if an English translation exists)
`url`	Source / canonical link to the document
`summary`	Short abstract of the policy content
`source.url`	URL actually fetched for the text snapshot
`source.retrieved`	Date the text snapshot was retrieved (YYYY-MM-DD)
`source.text_path`	Relative path to the extracted full-text file (`corpus/<NN>-slug.txt`)
`source.http_status`	HTTP status code returned when fetching the source
`source.bytes`	Size in bytes of the downloaded source file
`source.sha256`	SHA-256 checksum of the downloaded source (reproducibility)
`source.format`	Source file format: pdf / html
`source.status`	Text-extraction status: `ok` (full body) / `ok_landing` (landing-page text only)
`source.chars`	Number of characters in the extracted text
`source.original_url`	Original URL, preserved when the source was swapped for a better mirror
`source.via`	Retrieval route where non-standard (e.g. `ocr`, `wayback`, `curl`, `manual_pdf`)
`source.ocr_lang`, `source.snapshot_url`, `source.remediated`, `source.note`, `source.date_note`	Additional provenance where applicable

How the corpus was built

Documents were assembled by an AI-assisted retrieval protocol (query diversification and deduplication) over official and open-access legal, treaty, government, and basin-commission sources, followed by text extraction (pypdf for PDF, Beautiful Soup for HTML, OCR for scanned PDFs, and the Internet Archive Wayback Machine for relocated or blocked links). Every candidate was verified by human reviewers, and every text snapshot is frozen with a SHA-256 hash for reproducibility. Full details are in the accompanying data article.

Licence and reuse

—The harmonised metadata and the compilation in this repository are released under the Creative Commons Attribution 4.0 (CC BY 4.0) licence.
—The source document texts in corpus/ are reproduced as public official material. Official texts of a legislative, administrative, or legal nature are generally exempt from copyright (Berne Convention, Art. 2(4), and equivalent national provisions). Each record carries the source URL and issuing body so the original document can be attributed and verified. Copyright in any source text remains with its issuing authority.

Citation