AndreaBozzo/ceres-open-data-index

Ceres Open Data Index

A curated index of 2,745,021 open dataset records from 567 portal exports across 40+ countries and international sources, with cross-portal duplicates flagged. The snapshot includes normalized dataset metadata, resource links when supplied by the source, and a reproducible manifest with file checksums.

Dataset description

This dataset was harvested by Ceres, a harvest-first open-data toolkit. It covers ten source families: CKAN, DCAT-AP (udata REST), SPARQL-backed DCAT, Project Open Data (data.json), Socrata, OpenDataSoft, ArcGIS Hub, OGC Records/CSW, collection-level STAC, and SDMX. Noise is filtered during export; likely cross-portal duplicates are flagged but retained.

What's included

• —Titles, descriptions, URLs, tags, organizations, licenses, and source provenance
• —Resource-level names, formats, media types, URLs, and schema field counts when available
• —A heuristic cross-portal duplicate signal (is_duplicate)
• —Per-portal Parquet subsets for targeted analysis
• —A versioned manifest, quality report, and snapshot-to-snapshot changelog

What's not included

• —The data files themselves (CSVs, shapefiles, and similar assets)
• —Embedding vectors, which are model-specific
• —A canonical entity-resolution result; is_duplicate is only a title-based heuristic

Resource links are present for 1,997,796 rows (72.8%); 96,395 rows (3.5%) include at least one resource schema. Coverage varies substantially by source API.

Dataset structure

Schema

Files

• —all.parquet — complete dataset (2,745,021 rows)
• —data/<portal>.parquet — 567 portal subsets; rows repeat those in all.parquet
• —identity.parquet — record keys and content hashes used for snapshot diffs
• —metadata.json — snapshot manifest, Ceres provenance, curation counts, aliases, and SHA-256 checksums
• —reports.json / report.md — coverage and field-completeness report
• —changelog.json / changelog.md — added, changed, removed, and unchanged counts versus v5

Manifest paths are unique. This release contains 569 Parquet files in total (all.parquet, identity.parquet, and 567 portal subsets), and every file's size, checksum, and row count was verified before publication.

Largest portal exports

Top 48 by exported row count (of 567 total):

The full portal breakdown and exact source URLs are recorded in metadata.json and reports.json. Alias feeds such as a portal's root API and its /data.json endpoint can share a canonical export name without being treated as independent sources.

Coverage and quality

Exporter classification by source type: unmapped/legacy config 1,318,679 · DCAT 835,846 · CKAN 327,737 · OpenDataSoft 177,191 · OGC Records/CSW 27,264 · ArcGIS 24,302 · SDMX 15,160 · Socrata 11,833 · STAC 7,009. The unknown classification reflects missing type annotations in the export configuration, not unknown provenance; source URLs remain present.

DCAT profiles account for 817,318 SPARQL, 12,058 static JSON, and 3,477 udata REST records. Field completeness is: description 100.0% · resources 72.8% · modification date 52.4% · organization 51.2% · tags 45.3% · license 31.8% · resource schema 3.5%.

Language labels are source/portal metadata rather than per-row language detection. The largest labels are English (en) 1,182,023; unknown 1,161,936; eng 87,963; Spanish 76,992; French 75,086; Ukrainian 36,301; Czech 32,879; Greek 21,900; Japanese 14,224; Slovak 11,748; and Norwegian 8,669.

Loading the dataset

import pandas as pd

df = pd.read_parquet("all.parquet")
df_us = pd.read_parquet("data/catalog-data-gov.parquet")
likely_unique = df[~df["is_duplicate"]]

Or with DuckDB:

SELECT source_portal, count(*) AS records
FROM read_parquet('all.parquet')
GROUP BY source_portal
ORDER BY records DESC;

Dataset creation

Harvesting methodology

Ceres harvests each portal through its native API with a streaming, incremental pipeline:

1. 1.CKAN portals use paged package_search requests with adaptive page sizing and retry behavior.
2. 2.DCAT udata catalogs use paged JSON-LD; Project Open Data catalogs consume data.json documents.
3. 3.SPARQL-backed DCAT catalogs use stable keyset pagination, bounded distribution-property queries, and URI-level deduplication.
4. 4.Socrata uses the Discovery API; OpenDataSoft uses the Explore API with a keyset cursor beyond its 10,000-row window.
5. 5.ArcGIS Hub, OGC Records/CSW, collection-level STAC, and SDMX clients preserve family-specific source metadata.
6. 6.ceres export --format parquet streams PostgreSQL records, filters noise, flattens metadata, flags cross-portal title matches, and writes compressed Parquet plus checksummed reports.

The data.europa.eu refresh was intentionally stopped after resource enrichment reached approximately half the exported portal rows so this snapshot would not delay Ceres 0.7.0. The export contains 712,481 data.europa.eu records, of which 366,307 (51.4%) include resource metadata. Completing the remaining enrichment is tracked in Ceres issue #254 for the next release.

Curation

• —Noise filtering: 81,766 rows removed (short/noise titles and empty descriptions)
• —Duplicate flagging: 1,354,529 rows marked; records are retained and no canonical entity resolution is claimed
• —Alias normalization: known mirror or alternate endpoints can share a canonical portal identity
• —Metadata flattening: nested source fields and resource metadata are normalized into the published schema

The duplicate method and version are recorded under duplicate_detection in metadata.json.

Update and provenance

Snapshot `ceres-20260819-884e6da7b4b0` was generated on 2026-08-19 by Ceres 0.7.0 at commit `2583fcf`. Relative to ceres-20260715-ed0c345e8663, it has 183,625 added, 172,349 changed, 2,613 removed, and 2,389,047 unchanged records. See changelog.md for the per-portal breakdown.

Considerations

Known biases and limitations

• —Large aggregators dominate the index: data.europa.eu contributes 26.0% and catalog.data.gov 14.6% of rows.
• —The partial data.europa.eu refresh means resource coverage is materially better than v5 but not complete.
• —49.3% of rows are flagged as possible cross-portal title duplicates, especially because aggregators re-list member-portal records.
• —Metadata quality, vocabulary, timestamps, and license conventions vary by source; only 31.8% of rows have a non-empty license.
• —SPARQL, STAC, OGC, and SDMX records may have fewer normalized fields than CKAN or Socrata records.
• —Exporter language and source-type annotations are incomplete for legacy configurations, but original source URLs and metadata are retained.

Ethical considerations

This dataset contains public catalog metadata from government and institutional portals. It does not intentionally contain the underlying data files. Source licenses vary; consumers must follow the license and terms attached to each source record.

Additional information

License

This aggregate dataset and its tooling are released under Apache 2.0. Underlying source metadata may carry separate open-data licenses recorded per row where available.

Citation

@misc{ceres-open-data-index-2026,
  title={Ceres Open Data Index},
  author={Andrea Bozzo},
  year={2026},
  url={https://github.com/AndreaBozzo/Ceres},
  note={2,745,021 records from 567 portal exports, snapshot 2026-08-19}
}

Links

• —Ceres on GitHub
• —Ceres documentation