GPS-bench/gps-bench-ai-bills

GPS-Bench AI Legal Instruments Dataset

An evidence-backed corpus of global AI-related laws, regulations, policies, guidelines, standards, and strategic instruments. Each canonical record retains its official title, issuing authority, normalized jurisdiction, legal status, dates, primary-source citations, and full verified text. Every non-English instrument keeps its source-language original as the authoritative text, with an English translation beside it produced by a single translator, gemini-3.5-flash. The companion benchmark view projects instruments evaluated against expert human analyses.

This dataset card documents the canonical tables, evaluation benchmark splits, querying patterns, and schema invariants for the Hugging Face Hub release.

Table of Contents

1. 1.Primary Corpus vs. Benchmark View
2. 2.Instruments the Corpus Does Not Actually Hold
3. 3.Dataset Overview & Geopolitical Scopes
4. 4.Release Structure & Contents
5. 5.Canonical Record Schema
6. 6.Loading & Querying the Dataset
7. 7.Jurisdiction & Scope Invariants
8. 8.Translation Provenance
9. 9.Ingestion Pipeline: Adding New Instruments
10. 10.Validation & Reproducibility
11. 11.Changelog

Primary Corpus vs. Benchmark View

The dataset is published with two complementary tables designed for distinct use cases:

Record counts live in `manifest.json`, not on this card. The build regenerates that file and publish_to_hf.py gates it against the shipped rows; a count typed into prose has nothing checking it, and ages silently every time an instrument or analysis is ingested. Read counts from the manifest, or from the tables themselves.

Evaluation Splits in benchmark_view.jsonl

Each benchmark row maps an instrument to its evaluation split:

• —`before_cutoff`: Pre-cutoff enactments with $\ge 2$ expert analyses, for standard policy comprehension evaluation.
• —`after_cutoff`: Post-cutoff instruments, evaluated against models without data contamination.
• —`train`: Held-out training split and few-shot in-context exemplar set.
• —`one_analysis`: Instruments accompanied by a single verified expert legal commentary. At least one instrument in this split carries two analyses rather than one — read analysis_count rather than assuming the split name.

Group by split on benchmark_view.jsonl for the current size of each.

Both tables are guaranteed to remain in 100% lockstep: the automated build (make build) and validation (make validate / publish_to_hf.py) pipelines verify that every instrument carrying expert analyses is synchronized with benchmark_view.jsonl before release.

Instruments the Corpus Does Not Actually Hold

A census on 2026-08-20, extended on 2026-08-22, read the captured text of every record and found 57 instruments whose document is not the instrument it claims to be. Each is flagged in place: documents[].extraction_provenance.cleaning_actions carries the class, defect_note says what is wrong in plain language, and defect_basis records the measured evidence and which census pass found it.

Fifteen have since been replaced with the instrument's own text, so 47 instruments still carry a flag. The repaired records are identifiable by extraction_provenance.tool == "ingestion/replace_document.py", and each carries a note saying how long the document it displaced was. The rest are still flagged and not repaired: a flag is a statement that somebody looked, not that somebody fixed it.

The second pass found six the first missed, and they divide the same way: three hold the wrong document outright (an Innovation Zurich news index standing for the Canton of Zurich sandbox; a board-institute article standing for the SDAIA principles; a Hawai'i committee vote history standing for SB 2687), and three are the real text wrapped in whitehouse.gov or Algoritmekader site chrome. The first census scored records on one structural signal; these six were classed by reading the captured text, which is why they took a second pass to see.

How the fifteen were repaired. ingestion/replace_document.py swaps a misfiled document for the instrument's own text and refuses a replacement no longer than what it overwrites — a shorter document is the tell that the new URL is another status page. That guard is waived only where the incumbent is itself page furniture, because length stops being a proxy for substance when the thing being replaced is a navigation menu: boston.gov's stored document ran 14,371 characters of site menu against 12,341 characters of actual policy.

The clearest cases were three White House instruments. One held nothing but 5 Days of Wins / Official White House App / INVESTMENT ANNOUNCEMENTS and not a single occurrence of "shall"; loading the real memoranda and the Federal Register text took them from zero statutory clauses to 29, 11 and 9.

Separately: control bytes are gone, and lossy decodes are not. PDF extraction emitted the page structure of a document as C0 control bytes — U+000C between pages, U+0007 and U+0008 where a glyph table did not map, and NUL. They are invisible in a terminal, survive JSON, and reach a tokenizer as garbage, which matters because people train on this. They are now normalised at extraction (ingestion.primary_text.normalise_controls: page and record separators become newlines, every other control byte is dropped) and the existing corpus was swept; validation/audit_text_quality.py fails if any return.

U+FFFD is deliberately not repaired. It marks a byte that failed to decode — Ar<?>cle for "Article", de<?>ne for "define", O<?>Neal for "O'Neal" — and the character it replaced is unrecoverable. Guessing would put invented words into legal text. The affected documents are counted as lossy_decode_documents by that audit so they can be re-extracted from source.

What this does and does not corrupt. It corrupts the corpus's claim to hold these 51 instruments. It mostly does not corrupt what was mined from them: a navigation page contains no normative language, so the miner finds nothing in it. The defect is largely self-limiting — most text-absent instruments yield zero clauses, and gps:cn-na-pdngaxx-2017, the worst on every signal, yields zero.

The headline number to resist: roughly 3% of the statutory clauses in `GPS-bench/actor` come from a flagged instrument — but that figure is dominated by the polluted class, whose clauses are genuine. Only a couple of dozen clauses come from a text-absent or mangled instrument, and hand-reading every one of those put the actually-junk figure at 8 rows — four of them the literal site string "Links to said data may not be functional at this time." mined as a prohibition.

You can check this yourself: the actor layer's bill_key is this dataset's instrument_id without its gps: prefix, and the mapping is total in both directions.

To exclude flagged records:

FLAGS = {
    "flagged_wrong_document_not_instrument",
    "flagged_announcement_page_not_instrument",
    "flagged_mangled_extraction",
}  # add "flagged_boilerplate_polluted" only if navigation text would harm your task

def holds_its_instrument(record):
    return not any(
        set((d.get("extraction_provenance") or {}).get("cleaning_actions") or []) & FLAGS
        for d in record.get("documents") or []
    )

Dataset Overview & Geopolitical Scopes

The dataset classifies legal instruments across four mutually exclusive geopolitical scopes (entity_context.scope):

Jurisdiction Kinds (jurisdiction_kind)

Every instrument is issued under a specific jurisdiction_kind:

• —`country`: Sovereign nation-state (e.g. US, CN, GB, CA, JP, SG).
• —`subnational`: Subordinate territorial jurisdiction, such as a US state or Canadian province (e.g. US-CA, US-TN, CA-ON).
• —`supranational`: Reserved exclusively for the European Union (`EU`), reflecting its unique supranational legislative powers directly enforceable across member states.
• —`multilateral`: Intergovernmental organizations, international treaties, declarations, and regional/bilateral partnerships (e.g. ASEAN, African Union, UN, OECD, G7, COE, GPAI, BRICS, Canada–EU MoU).

Release Structure & Contents

datasets/bill_analysis_dataset/
├── hf/                                  # Canonical published release
│   ├── instruments.jsonl                # Primary corpus: 1 row per canonical instrument
│   ├── benchmark_view.jsonl             # Benchmark projection: instruments paired with analyses
│   ├── schema/
│   │   ├── instruments.schema.json      # Authoritative JSON Schema for instruments.jsonl
│   │   └── benchmark_view.schema.json   # Authoritative JSON Schema for benchmark_view.jsonl
│   ├── manifest.json                    # Build metadata, sha256 checksums, and record counts
│   └── README.md                        # Hub dataset card (synced from root README.md)
├── mappings/                            # Controlled vocabularies and host mappings
│   ├── jurisdiction_mapping.yaml        # Organization codes, names, and country aliases
│   ├── source_type_mapping.yaml         # URL domain classification rules
│   └── status_mapping.yaml              # Regex pattern rules for status normalization
├── ingestion/                           # Ingestion tools (URL fetching, PDF parsing, translation)
│   ├── add_instrument.py                # fetch and add CLI subcommands
│   ├── primary_text.py                  # HTML/PDF cleaners and language detector
│   ├── refresh_summaries.py             # Generates grounded ai_relevance summaries
│   └── sync_release.py                 # Merges verified additions and syncs hf/ release
├── validation/                          # Integrity and quality audit scripts
│   ├── validate_dataset.py              # Cross-field relational and schema validator
│   ├── audit_text_quality.py            # Checks for tag leaks, boilerplate, and text lengths
│   └── deduplicate_dataset.py           # Multi-heuristic duplicate detector
├── schema.py                            # Pydantic v2 source of truth for all schemas
├── Makefile                             # Ingestion and build target orchestrator
└── README.md                            # Canonical documentation (this file)

Canonical Record Schema

The principal top-level fields in hf/instruments.jsonl are:

Annotation tasks

Each annotations[] row is {annotation_id, task, label, basis, annotator, created_at, confidence, runner_up}. The tasks currently written:

legislative_stage is derived by legislative_stage.py. For every US federal bill (114th–119th Congress) the stage comes from a govinfo BILLSTATUS audit of the bill's own action history, validated against govinfo's structured fields (the <laws> element for enactment; Engrossed text versions for chamber passage) with zero mismatches; veto-override enactments outrank the veto. Bills of an ended Congress that were not enacted are dead, with the furthest stage reached preserved in the basis ("reached passedchamber, then died with the 117th Congress"). For all other instruments the stage is derived from stored status fields and is an **ingestion-time snapshot** — every row's `basis` ends with an `asof marker (final, audited YYYY-MM-DD, or ingestion snapshot) saying which; soft-law and executive instruments with no legislative pipeline are not_applicable`.

industry and policy_topic are two independent axes produced by a deterministic lexicon classifier with a proper-noun guard and an evidence floor (sector_classification.py). Every row's basis carries the verbatim evidence sentence from the stored document text and its locator, so a label can be checked without rerunning anything; confidence is the label's share of the instrument's lexicon hits. Only corroborated labels are written — uncorroborated candidates stay out of the release. government (industry: bills about the public sector) is distinct from government_use_of_ai (topic: bills governing agencies' own use of AI). Labels are machine annotations, named as such in annotator, and coexist with the human single-label coverage_area.

Loading & Querying the Dataset

Using Python & Hugging Face datasets

from datasets import load_dataset

# Load canonical legal instruments
ds = load_dataset("GPS-bench/gps-bench-ai-bills", "instruments", split="train")

# 1. Filter by jurisdiction kind (country, subnational, supranational, multilateral)
countries_only = ds.filter(lambda row: row["jurisdiction_kind"] == "country")
multilateral_only = ds.filter(lambda row: row["jurisdiction_kind"] == "multilateral")

# 1b. Filter by industry / policy-topic annotation (multi-label, evidence-backed)
def has_label(row, task, label):
    return any(a["task"] == task and a["label"] == label for a in row["annotations"])

healthcare_bills = ds.filter(lambda row: has_label(row, "industry", "healthcare"))
transparency_bills = ds.filter(lambda row: has_label(row, "policy_topic", "transparency_disclosure"))

# 2. Filter by geopolitical scope (us, china, other_countries, multilateral)
other_countries = ds.filter(lambda row: row["entity_context"]["scope"] == "other_countries")
us_instruments = ds.filter(lambda row: row["entity_context"]["scope"] == "us")

# 3. Filter by specific country code (ISO 3166-1 alpha-2)
uk_bills = ds.filter(lambda row: row["country_code"] == "GB")
canada_bills = ds.filter(lambda row: row["country_code"] == "CA")

# 4. Combined filter: Enacted national laws across select countries
target_countries = {"CA", "GB", "JP", "SG", "AU"}
enacted_laws = ds.filter(
    lambda row: row["country_code"] in target_countries
    and row["jurisdiction_kind"] == "country"
    and row["status_class"] == "enacted"
    and row["normative_character"] == "hard_law"
)

# Load benchmark view (instruments paired with human analyses)
benchmark = load_dataset(
    "GPS-bench/gps-bench-ai-bills", "benchmark_view", split="train"
)

High-Performance Querying with Polars

import polars as pl

df = pl.read_ndjson("hf/instruments.jsonl")

# 1. Filter by jurisdiction_kind
subnational_df = df.filter(pl.col("jurisdiction_kind") == "subnational")
multilateral_df = df.filter(pl.col("jurisdiction_kind") == "multilateral")

# 2. Filter by geopolitical scope (entity_context.scope)
other_countries_df = df.filter(
    pl.col("entity_context").struct.field("scope") == "other_countries"
)

# 3. Filter by country code (single or multiple ISO codes)
uk_df = df.filter(pl.col("country_code") == "GB")
g7_df = df.filter(pl.col("country_code").is_in(["US", "GB", "CA", "FR", "DE", "IT", "JP"]))

# 4. Combined filter: Hard-law enactments outside US & China with summaries
filtered = (
    df.filter(
        (pl.col("entity_context").struct.field("scope") == "other_countries")
        & (pl.col("jurisdiction_kind") == "country")
        & (pl.col("normative_character") == "hard_law")
        & (pl.col("status_class") == "enacted")
    )
    .select(["instrument_id", "title_official", "country_code", "status_class", "ai_relevance"])
)
print(filtered)

# 5. Summary statistics: count instruments by scope and status
summary = (
    df.group_by(
        [
            pl.col("entity_context").struct.field("scope").alias("scope"),
            pl.col("jurisdiction_kind"),
            pl.col("status_class"),
        ]
    )
    .agg(pl.len().alias("count"))
    .sort(["scope", "count"], descending=[False, True])
)
print(summary)

Memory-Efficient Streaming with Standard Python

import json

with open("hf/instruments.jsonl", encoding="utf-8") as f:
    for line in f:
        row = json.loads(line)
        scope = row["entity_context"]["scope"]
        kind = row["jurisdiction_kind"]
        country = row["country_code"]

        # Filter for non-US/China sovereign country laws
        if scope == "other_countries" and kind == "country" and country == "JP":
            print(f"Japan: {row['instrument_id']} — {row['title_official']}")
        elif kind == "supranational":
            print(f"Supranational: {row['instrument_id']} ({row['title_official']})")
        elif kind == "multilateral":
            print(f"Multilateral:  {row['instrument_id']} ({row['jurisdiction']})")

Jurisdiction & Scope Invariants

When querying, filtering, or contributing new instruments, keep these core invariants in mind:

1. 1.Supranational vs. Multilateral:
2. 2.jurisdiction_kind == "supranational" is strictly limited to the European Union (`EU`).
3. 3.All other international organizations (ASEAN, African Union, G7, OECD, UN, GPAI, Council of Europe, BRICS, bilateral MoUs) must use jurisdiction_kind == "multilateral".
4. 4.Entity Scope (`entity_context.scope`):
5. 5.us: US federal + state/local instruments.
6. 6.china: China national + provincial instruments.
7. 7.other_countries: Sovereign nations and territories (UK, Canada, Japan, etc.).
8. 8.multilateral: Multilateral organizations and the EU.
9. 9.Regime Annotations:
10. 10.Multilateral and supranational entities receive regime: "NA" with the basis "Not applicable to a supranational or multilateral entity." because V-Dem democracy indices only evaluate sovereign nation-states.

Translation Provenance

Non-English instruments carry two documents: the source-language original, which is the authoritative text, and an English rendering beside it. The English rendering is never the instrument — it exists for search, retrieval, and analysis.

Every machine translation in this release was produced by one model, `gemini-3.5-flash`. Earlier revisions accumulated renderings from six different models, because each ingestion pass used whichever model that pass happened to reach for. That is a benchmark defect rather than a cosmetic one: when the English text of two instruments comes from two different translators, a disagreement measured downstream cannot be attributed to the instruments rather than to their translators. Re-rendering the whole corpus through a single translator removes that confound.

Counts are in manifest.json, not here — this card cannot be regenerated and its numbers went stale once already.

The model is recorded in `extraction_provenance.tool`, which is the field the schema reserves for it — ExtractionProvenance is closed (additionalProperties: false) and has no model field. Read the translator from tool, not from note.

Analyses carry the same distinction, in `analyses[].full_text_en_machine`. An analysis published in Latvian, Danish, Spanish or French keeps its excerpt and full_text in the source language — the quote gate re-fetches the cited page and requires the excerpt verbatim, so a translated excerpt would be refused, and rightly: it is not what the page says. The English rendering travels beside them for the readers downstream that only have English lexicons. Measured 2026-08-22, a Latvian analysis of a real 2024 deepfake statute produced nothing at all until it had one. As with instrument text, the translation is model output and is never the authority for what the analysis says.

The two official documents are English texts published by the issuing government itself (Israel's AI Policy 2023 and Government Resolution 173). They are not model output and are deliberately excluded from re-translation. Note that Israel's AI Policy states on its own page 3 that it is an English summary of the Hebrew policy, not a full translation; the Hebrew remains the complete instrument.

Machine translations cover 18 source languages, most commonly Spanish (18 instruments) and Chinese (11). Run make audit-models to reproduce the table above from the release itself.

Ingestion Pipeline: Adding New Instruments

Ingesting new primary sources uses a strict two-step URL-first workflow via ingestion/add_instrument.py:

Step 1: Fetch & Inspect (fetch)

make ingest ARGS="fetch --url 'https://example.gov/official-ai-act'"

• —Downloads the official source (HTML or PDF via Poppler).
• —Automatically detects language and generates a checkpointed gemini-3.5-flash English translation if non-English.
• —Performs duplicate detection against existing canonical records.
• —Caches raw output under inputs/raw/fetched/<slug>.json for human inspection.

Step 2: Build & Validate (add)

Review the fetched text and run add (dry run first, then --write):

make ingest ARGS="add \
  --url 'https://example.gov/official-ai-act' \
  --instrument-id gps:ca-act-2026 \
  --title 'Official English Title' \
  --issuer 'Ministry of Industry' \
  --jurisdiction 'Canada' \
  --jurisdiction-kind country \
  --country-code CA \
  --instrument-type act \
  --status-raw 'Royal Assent' \
  --status-class enacted \
  --normative-character hard_law \
  --issued-date 2026-03-01 \
  --ai-relevance 'Establishes safety testing duties for frontier AI models.' \
  --write"

For multilateral or non-country instruments:

make ingest ARGS="add \
  --url 'https://asean.org/example-guide.pdf' \
  --instrument-id gps:asean-sample-guide-2026 \
  --title 'ASEAN Sample AI Guide' \
  --issuer 'ASEAN Secretariat' \
  --jurisdiction 'Association of Southeast Asian Nations' \
  --jurisdiction-kind multilateral \
  --organization-code ASEAN \
  --instrument-type guideline \
  --status-raw 'Adopted' \
  --status-class guidance \
  --normative-character soft_law \
  --issued-date 2026-05-10 \
  --ai-relevance 'Regional framework for trustworthy AI deployment.' \
  --write"

Validation & Reproducibility

To maintain integrity across the release, standard Makefile targets are provided:

make build       # Synchronizes verified additions from inputs/ into hf/ and updates README
make validate    # Validates cross-field schemas, unique IDs, and entity context
make audit-text  # Audits document completeness, HTML leaks, and translation coverage
make test        # Runs full test suite (validate + audit-text)
make actor       # Rebuilds actor relationship graph and explorer map
make explorer    # Rebuilds the standalone HTML Bill Explorer
make deduplicate # Runs multi-heuristic duplicate check across the corpus
make summaries   # Regenerates grounded English summaries for verified additions

Changelog

Data revision 2026-08-21

• —Defective captures flagged, not hidden: a census read every captured document and flagged 51 instruments whose text is not the instrument they claim to hold — 27 absent, 3 mangled, 21 boilerplate-polluted. See Instruments the Corpus Does Not Actually Hold. extraction_provenance gained defect_note and defect_basis to carry the finding and its evidence.
• —Benchmark view grew substantially, as newly ingested expert analyses were paired with their instruments. Every split changed size. Hard counts have since been removed from this card in favour of manifest.json, which the build keeps current.

Data revision 2026-08-20

• —US federal AI executive orders completed: added EO 14110 (Safe, Secure, and Trustworthy Development and Use of AI, 2023), EO 13960 (Trustworthy AI in the Federal Government, 2020) and EO 14141 (AI Infrastructure, 2025) with full Federal Register text. The corpus previously jumped from EO 13859 (2019) straight to 2025, leaving the whole 2020-2024 span of US executive action on AI unrepresented.
• —Single translator across the corpus: re-rendered every machine translation with gemini-3.5-flash, replacing renderings previously produced by gpt-4.1, deepseek-v4-pro, claude-sonnet-4-5, and gemini-3.6-flash. The model is now recorded uniformly in extraction_provenance.tool.
• —Vietnam AI Law rebuilt from the Official Gazette: gps:vn-ai-law-2025 previously held an OCR of a scanned signature PDF plus an English scrape of a commercial legal database that was mislabelled official. Both were replaced from the Công báo DOCX (Articles 1–35, Chapters I–VIII).
• —Record counts corrected: the card had been understating the release, and the counts it carried had drifted from the build. They now live in manifest.json instead.

Version 2.3.0 (2026-08)

• —Updated Geopolitical Scopes: Standardized entity_context.scope to four mutually exclusive scopes: us, china, other_countries, and multilateral.
• —Multilateral Jurisdiction Kind: Renamed intergovernmental to multilateral under jurisdiction_kind.
• —EU Supranational Rule: Restated supranational exclusively for European Union (EU) instruments, classifying all other international bodies (ASEAN, African Union, UN, OECD, G7, COE, GPAI, BRICS, bilateral MoUs) under multilateral.
• —Verified Primary Sources: Added verified primary text extractions, schema validation rules, and comprehensive dataset documentation.