dean22029/WTO_Docs

WTO Dispute Settlement Body Documents

Full-text corpus of official WTO Dispute Settlement Body (DSB) documents spanning DS1–DS626, covering January 1995 through early 2026. Sourced from the WTO's public case repository and processed into structured records for retrieval-augmented generation (RAG) and NLP research.

Coverage

Data Structure

Each line in wto_documents_full.jsonl is a JSON object with the following fields:

Case-level fields (repeated per document within a case)

Document-level fields

Document Types

The 42 consolidated document types, with record counts:

Dispute Stages

The dispute_stage field reflects the furthest procedural stage reached by each case:

Text Cleaning

The clean_text field has been processed through a 10-step pipeline optimized for embedding quality:

1. 1.Header-area boilerplate removal (WTO cover page patterns)
2. 2.Document codes stripped (WT/DS..., G/...)
3. 3.Language markers removed (anglais/English/français)
4. 4.Page numbers removed
5. 5.Footnotes removed (underscore separators + numbered continuations)
6. 6.Non-English lines removed (French/Spanish detected via function-word threshold)
7. 7.Repeated case titles deduplicated
8. 8.Punctuation noise cleaned
9. 9.Whitespace normalized

OCR (Tesseract) was applied as a fallback for 78 scanned PDFs where PyPDF extraction yielded fewer than 50 characters.

Usage Example

With the HuggingFace datasets library (recommended)

from datasets import load_dataset

# Load full dataset
ds = load_dataset("dean22029/WTO_Docs")
df = ds["train"].to_pandas()

# Filter to a single case
ds267 = ds["train"].filter(lambda x: x["case_number"] == "267")

# Filter by document type
consultations = ds["train"].filter(
    lambda x: x["doc_type"] == "Request_For_Consultations"
)

# Filter by dispute stage
appellate = ds["train"].filter(
    lambda x: x["dispute_stage"] == "Appellate Body"
)

# Access a record
print(ds["train"][0]["case_number"])   # "1"
print(ds["train"][0]["doc_type"])      # "Request_For_Consultations"
print(ds["train"][0]["clean_text"][:300])

With plain Python (no dependencies)

import json

with open("wto_documents_full.jsonl") as f:
    for line in f:
        doc = json.loads(line)
        print(doc["case_number"], doc["doc_type"], doc["date"])
        print(doc["clean_text"][:300])
        break

Load all documents for a specific case:

import json

def get_case_docs(jsonl_path, case_number, doc_type=None):
    docs = []
    with open(jsonl_path) as f:
        for line in f:
            doc = json.loads(line)
            if doc["case_number"] == str(case_number):
                if doc_type is None or doc["doc_type"] == doc_type:
                    docs.append(doc)
    return docs

# All documents for DS267 (EC - Beef Hormones)
all_docs = get_case_docs("wto_documents_full.jsonl", 267)

# Consultation requests only
consultations = get_case_docs("wto_documents_full.jsonl", 267, "Request_For_Consultations")

Parse the `third_parties` field:

import ast

doc = json.loads(line)
third_parties = ast.literal_eval(doc["third_parties"])  # e.g. ["USA", "Canada"]

Data Source and Processing

Documents were scraped from the WTO Dispute Settlement Gateway using Selenium. PDFs were parsed with PyPDFLoader (text-based) and Tesseract OCR (scanned). Dates were extracted multilingually (English, French, Spanish) from PDF headings.

Case metadata (complainant, respondent, third parties, dispute stage, agreements cited, case summary) was scraped separately from WTO case pages and joined by case number.

Known Limitations

• —third_parties field stores Python list repr strings (e.g. "['USA', 'EU']"); parse with ast.literal_eval().
• —date is null for ~4.5% of records (mostly untitled addenda and cross-reference files).
• —Non-English documents (primarily French/Spanish originals) have reduced clean_text quality after line-level language filtering.
• —Taiwan (Chinese Taipei) has no UN ideal point data in linked panel datasets — expected, as it is not a UN member.
• —DS627+ cases exist in case metadata but have no associated PDFs in this corpus (collection cutoff: DS626).