GreatNorthCollective/greatnorth-ai-register

Great North AI Register

A normalized, reproducible dataset of public-sector AI system metadata from the Government of Canada AI Register (Minimum Viable Product), prepared by Great North Collective.

Dataset Description

This dataset provides a clean, machine-readable snapshot of the Government of Canada AI Register as published on Open Canada. It contains structured records for AI systems used or developed by federal government organizations, including system names, responsible departments, descriptions, status, and related operational details.

The data is intended for:

• —Sovereign AI research and North American AI governance studies
• —Civic transparency and public-sector AI inventory work
• —Retrieval-augmented generation (RAG) over government AI systems
• —Evaluation of public-sector AI disclosure practices
• —Training and benchmarking of models on real administrative metadata

Row count (as of build): 412 records Source snapshot date: See retrieved_at field in each record (ISO 8601 UTC)

Source Data

• —Primary source: Government of Canada AI Register (Minimum Viable Product) on Open Canada (CKAN)
• —Package ID: fcbc0200-79ba-4fa4-94a6-00e32facea6b
• —Selected resource: CSV (bilingual English/French) – the most recent machine-readable export available at fetch time
• —Upstream licence: Open Government Licence – Canada (see source_license per record)
• —Attribution: Contains information licensed under the Open Government Licence – Canada.

The pipeline fetches live package metadata via the CKAN Action API, selects the preferred CSV resource, downloads it, and normalizes it without any LLM calls or external enrichment.

Data Fields

See the full data dictionary for every field, type, nullability rules, and derivation logic.

Key stable fields:

• —id: Deterministic Great North Collective identifier (gnc-ai-register-*)
• —system_name, department, description, status
• —uses_ai: true for all rows (by construction)
• —source_url, source_resource_url, retrieved_at, source_license
• —raw_record: Complete original row as compact JSON (full audit trail)

All 24 original source columns are preserved inside raw_record for every row.

Intended Uses

• —Research on public-sector AI transparency and disclosure patterns
• —Building knowledge graphs, RAG corpora, or evaluation suites for government AI systems
• —Training or fine-tuning models that need realistic administrative AI metadata
• —Policy analysis and civic technology projects focused on North America
• —Baseline for comparing Canadian federal AI inventory practices with other jurisdictions

Out-of-Scope Uses

• —Real-time production decision systems (this is a periodic snapshot)
• —Any use that implies the dataset constitutes complete, authoritative, or exhaustive government-wide coverage (it is the published MVP register only)
• —Automated scoring or risk classification of individual systems without reference to the original source and context
• —Training models that generate synthetic government decisions or personal information

Collection Process

1. 1.CKAN package_show action call (primary + /en/ fallback) using the fixed package ID.
2. 2.Resource selection: CSV > JSON > XLSX/XLS, with language preference for English/bilingual.
3. 3.Raw download to data/raw/.
4. 4.Flexible column mapping (case-insensitive, multiple candidate names per target field).
5. 5.Deterministic ID generation + required field population (country="Canada", jurisdiction="Federal", uses_ai=true).
6. 6.Emission of JSONL (one record per line) + CSV with stable column order.
7. 7.Validation (row counts, unique IDs, provenance fields, JSON round-tripping, invariants).
8. 8.Dataset card + schema generation.

No rows are dropped. No fields are synthesized. All transformations are deterministic and logged via retrieved_at + full raw_record.

Licensing and Attribution

• —This derived dataset (scripts, schema, card, normalization logic): Licensed under the terms chosen by Great North Collective for the repository (currently marked other on the Hub pending formal licence decision).
• —Source data: Preserves upstream licence metadata per record. The large majority of records carry "Open Government Licence – Canada".
• —Attribution requirement (source): "Contains information licensed under the Open Government Licence – Canada."

When using this dataset, cite both the original source and this normalized version.

Limitations

• —The source is explicitly labeled "Minimum Viable Product" (MVP) by the Government of Canada. Coverage, completeness, and field population are not guaranteed to be government-wide or exhaustive.
• —Many records have incomplete fields (status, dates, capabilities, etc.). This is expected and preserved.
• —Dates (status_date) are often low-precision (year only) or appear to represent planned rather than historical dates.
• —Bilingual content exists in the source; the normalized view prefers English fields while keeping everything in raw_record.
• —No risk, domain, or AIA fields are present in the current source snapshot; these columns are null by design.
• —The register may evolve; this pipeline produces reproducible snapshots, not a live API.

Reproducibility

# One-command local build (after cloning)
python -m greatnorth_ai_register.cli all

# Or step-by-step
python -m greatnorth_ai_register.cli fetch
python -m greatnorth_ai_register.cli build
python -m greatnorth_ai_register.cli validate

• —Raw snapshot: data/raw/package_metadata.json + downloaded CSV
• —Processed outputs: data/processed/greatnorth_ai_register.{jsonl,csv}
• —Schema: docs/schema.md
• —Environment: see pyproject.toml / requirements.txt (Python ≥ 3.9)

To upload the dataset to the Hub (after setting HF_TOKEN):

python -m greatnorth_ai_register.cli upload --repo-id GreatNorthCollective/greatnorth-ai-register

Citation

Preferred citation:

Great North Collective. (2025). Great North AI Register (Version 0.1.0) [Dataset]. Hugging Face. https://huggingface.co/datasets/GreatNorthCollective/greatnorth-ai-register

Source attribution:

Contains information licensed under the Open Government Licence – Canada. Source: Government of Canada AI Register (MVP), Open Canada CKAN package fcbc0200-79ba-4fa4-94a6-00e32facea6b.

Contact & Collaboration

This is the first public dataset release from Great North Collective – an open sovereign AI collaboration for North America.

• —Issues / contributions: https://github.com/GreatNorthCollective/greatnorth-ai-register (or the org)
• —Sovereign AI focus: public-sector transparency, reproducible data pipelines, and North American data sovereignty.

We welcome high-quality, inspectable, non-hallucinated contributions that improve coverage, validation, or documentation while respecting the source.