Datasetpublic

NoeFlandre/osm-polygon-wikidata-and-wikipedia

OSM Polygon Wikidata + Wikipedia, V2 V2 builds on the V1 Wikidata-only dataset with multilingual Wikipedia and Wikivoyage text. It also retains valid multilingual wikipedia=* references, including polygons without a Wikidata QID. Source code: GitHub repository. Dataset snapshot Metric Value Polygon rows across regional extracts 1,259,424 Unique polygon identities (osm_type, osm_id) 1,188,854 Polygons with successful non-empty text (unique OSM… See the full description on the dataset page: https://huggingface.co/datasets/NoeFlandre/osm-polygon-wikidata-and-wikipedia.

sourceHugging Faceodblupdated 4d agoView on Hugging Face

0likes3.8kdownloads

Dataset Card

[image]

OSM Polygon Wikidata + Wikipedia, V2

V2 builds on the V1 Wikidata-only dataset with multilingual Wikipedia and Wikivoyage text. It also retains valid multilingual wikipedia=* references, including polygons without a Wikidata QID.

Source code: GitHub repository.

Dataset snapshot

Metric	Value
Polygon rows across regional extracts	1,259,424
Unique polygon identities (osmtype, osmid)	1,188,854
Polygons with successful non-empty text (unique OSM identities)	684,411
Wikipedia + Wikivoyage documents	2,332,127
Wikipedia + Wikivoyage sections	12,666,253
Wikipedia + Wikivoyage languages	353
Geographic regions	386
Total Parquet size	21.2 GB

Polygon rows preserve regional records. Identity metrics use one deterministic representative per (osm_type, osm_id). Text requires fetch_status=ok and a trimmed non-empty full_text.

Text volume: 833,590,596 document words; 74,791,929 sentence rows.

Document words count full Wikipedia and Wikivoyage document text; section rows are excluded. Sentence rows include split and explicitly unsplit unsupported-language rows when sentence sidecars exist.

Sentence-splitting coverage

Measured over input text units (sections), not output sentence rows.

—Eligible text units: 12,666,253
—Units split because language is supported: 11,197,076
—Units left unsplit because language is unsupported: 1,469,177
—Supported-language coverage: 88.4%
—Unsupported-language share: 11.6%
—Top unsupported languages by count: arz (277,524); ce (178,753); tt (97,105); sh (58,760); hr (43,936); nn (37,168); zh-min-nan (36,499); ast (36,246); azb (31,494); simple (26,509)

Coverage maps

All polygon identities

[image]

One point per unique retained polygon identity, including identities without text.

Polygon identities with text

[image]

One point per unique identity with successful non-empty text (684,411).

H3 text density

[image]

Each H3 cell shows the absolute count of unique text-covered polygon identities; it is not a proportion.

Geographic distribution by continent

Counts use one deterministic representative per polygon identity and the same text definition as the maps.

Continent	Polygons	Wikipedia documents	Wikivoyage documents	Polygons with Wikipedia text	Polygons with Wikipedia or Wikivoyage text	Text coverage
Africa	11,580	55,726	550	8,580	8,580	74.1%
Antarctica	264	1,724	1	262	262	99.2%
Asia	123,435	393,903	1,811	100,603	100,623	81.5%
Europe	790,203	1,087,427	5,163	384,442	384,442	48.7%
North America	161,918	395,992	3,429	120,410	120,411	74.4%
Oceania	20,149	30,960	396	11,825	11,826	58.7%
South America	19,771	64,103	786	14,287	14,287	72.3%
Unassigned	61,534	194,944	2,007	43,978	43,980	71.5%

Polygon area and geometry

Recorded polygon area: 181,690,954.0535 km² total; median 1,301.0 m². Complete area distribution, histogram, geometry, extent, and per-source statistics: `stats.json`.

Schema

Dataset Viewer schema. The published tables are documented by their Dataset Viewer configurations.

Data sources & licenses

OpenStreetMap polygons are ODbL; Wikidata is CC0; Wikipedia and Wikivoyage text are CC BY-SA 4.0.

How to load

python

from datasets import load_dataset
ds = load_dataset("parquet", data_files={"polygons": "hf://datasets/NoeFlandre/osm-polygon-wikidata-and-wikipedia/polygons/*.parquet"})

Citation

Download the dataset citation metadata from `CITATION.cff`.

Reproducibility

Run V2 explicitly with sync-dir --dataset-version v2. V1 remains a separate contract and is not modified by V2 processing.

Language partitions

The V2 language release is an additive, row-level partition of the published text tables. Each source row is routed by its normalized language value; multilingual rows are not collapsed to a polygon-level preferred language. Missing, blank, malformed, and legacy-unusable values are preserved in the explicit lang-unknown partition.

Validated languages: 350 (including unknown).

Configuration	Split names	Remote path
`polygon_document_links_by_language`	`polygon_document_links_by_language__lang_<language>` (split `train`) and `polygon_document_links_by_language__lang_unknown` (split `train`)	`language_splits/polygon_document_links_by_language/lang-<language>/part-*.parquet`
`wikipedia_documents_by_language`	`wikipedia_documents_by_language__lang_<language>` (split `train`) and `wikipedia_documents_by_language__lang_unknown` (split `train`)	`language_splits/wikipedia_documents_by_language/lang-<language>/part-*.parquet`
`wikipedia_sections_by_language`	`wikipedia_sections_by_language__lang_<language>` (split `train`) and `wikipedia_sections_by_language__lang_unknown` (split `train`)	`language_splits/wikipedia_sections_by_language/lang-<language>/part-*.parquet`

The release manifest records the source fingerprint, schema, row counts, and SHA-256 hash for every generated file.