Datasetpublic

NoeFlandre/osm-polygon-website-tag-eunis

OSM Polygon Website Dataset OpenStreetMap closed ways and polygon relations carrying a non-empty website OR contact:website tag, with full main-page text extracted using Trafilatura. Every statistic below is regenerated from the current upload-acknowledged Parquet artifacts. Snapshot Metric Value What it means Snapshot status In progress Current published snapshot Regional PBFs included 386 / 386 Published source shards / expected source PBFs… See the full description on the dataset page: https://huggingface.co/datasets/NoeFlandre/osm-polygon-website-tag-eunis.

sourceHugging Faceodblupdated 5d agoView on Hugging Face

0likes53downloads

Dataset Card

OSM Polygon Website Dataset

[image]

OpenStreetMap closed ways and polygon relations carrying a non-empty website OR contact:website tag, with full main-page text extracted using Trafilatura. Every statistic below is regenerated from the current upload-acknowledged Parquet artifacts.

Snapshot

Metric	Value	What it means
Snapshot status	In progress	Current published snapshot
Regional PBFs included	386 / 386	Published source shards / expected source PBFs
Published polygon rows	1,726,474	Rows in the public `polygons/` files
Comparison observations	2,716,855	Source-level records with a website, contact:website, or Wikidata tag
Duplicate OSM objects	104,927	Objects observed in more than one source snapshot
Conflicting snapshot observations	15	Repeated observations whose tag values disagree with the selected version
Rejected polygon candidates	868,905,743	Candidate objects that did not produce a usable polygon row

Website text

Tag	URLs	Successful	Empty	Failed	Words
`website`	1,518,605	1,075,910	56,248	386,447	368,299,585
`contact:website`	169,238	123,520	4,935	40,783	39,386,070

Website-text table counts are unique (osm_type, osm_id) identities across regional rows; regional overlap duplicates are removed globally.

Unique polygons with extracted text: 1,192,980 Counts unique (osm_type, osm_id) polygons across regional rows when any copy has successful, trimmed non-empty website or contact:website text; regional overlap duplicates removed globally. Combined extracted words: 407,685,655

Languages

Detected with GlotLID v3 on successfully extracted text; labels are exact script-aware language_Script codes with a top-1 probability column.

Metric	Value
Distinct languages	397
Labeled `website` texts	1,075,910
Labeled `contact:website` texts	123,520

Top 10 labels across both tags:

Language	Texts
`eng_Latn`	533,896
`deu_Latn`	216,762
`fra_Latn`	85,516
`pol_Latn`	42,731
`nld_Latn`	30,533
`spa_Latn`	30,358
`rus_Cyrl`	28,691
`ita_Latn`	25,038
`jpn_Jpan`	24,208
`por_Latn`	22,377

Sentences

Extracted text is segmented with SaT for the 85 languages the segmenter covers; text in any other detected language records unsupported_language instead of sentences. Segments carry their own trailing spaces but not the line breaks that separated them, so joining them does not reproduce the source text; the full text stays in the *_text columns.

Metric	Value
Sentences	45,823,447
Segmented `website` texts	1,073,468
Segmented `contact:website` texts	125,132
Texts in an uncovered language	37,473

Mean sentences per segmented text: 38.2

Polygon geometry

Surface and shape statistics computed over every published polygon row from the area_m2, bbox, and geometry columns. Areas are geodesic on the WGS84 ellipsoid. Population scope: published polygon rows. The complete breakdown is published as `stats.json`.

Metric	Value
Polygons measured	1,726,474
Total area	906,230.67 km²
Median area	946.58 m²
Mean area	524,902.59 m²
Smallest / largest area	0.01 / 129,636,243,089.97 m²
p95 area	82,053.29 m²
MultiPolygon rows	32
Rows with holes	20
Rows below 1 m²	393

Dataset bounding box: [-179.992396, -77.847199, 179.325994, 82.172551] (min lon, min lat, max lon, max lat).

Geographic distribution

[image]

H3 resolution 3 contains 5,850 occupied cells across 1,192,980 unique polygons with successfully extracted, non-empty website or contact:website text, globally deduplicated by (osm_type, osm_id); regional overlap duplicates removed globally. The color scale is logarithmic, counts are absolute, and a Natural Earth 1:110m land backdrop provides geographic context.

Links

Live metrics: Trackio dashboard; it shows this frozen dataset snapshot. Code and README: GitHub repository and README.

Top `website` hostnames

Hostname	Polygons
`www.facebook.com`	10,238
`www.mcdonalds.com`	9,886
`tools.usps.com`	6,382
`www.dollargeneral.com`	5,909
`locations.tacobell.com`	4,579
`www.walmart.com`	4,446
`www.hilton.com`	4,366
`www.marriott.com`	4,314
`www.cvs.com`	3,975
`www.bk.com`	3,806

Top `contact:website` hostnames

Hostname	Polygons
`www.7-eleven.com`	1,587
`www.tesco.com`	1,441
`5ka.ru`	1,224
`rosneft-azs.ru`	848
`www.mcdonalds.com`	827
`locations.dunkindonuts.com`	744
`storelocator.asda.com`	692
`lukoil.ru`	575
`www.netto-online.de`	498
`www.pochta.ru`	497

Methodology and quality

Geometry is assembled with libosmium. Full main text is extracted independently for both website tags with Trafilatura and is not truncated. Word counts are Python Unicode \w+ matches.

Text statuses are absent, pending, success, empty, invalid_url, unsafe_url, fetch_error, or extract_error. A source is enriched only when every status is success or absent. Failed values retry on later resumptions; successful values are cached.

A URL is marked unsafe_url when its hostname, or any redirect target, does not resolve exclusively to globally routable public IP addresses. Localhost, private, reserved, multicast, and unspecified targets are blocked. Unsupported schemes and URLs containing credentials are classified as invalid_url; redirect limits, timeouts, oversized responses, and unsupported content types are recorded as fetch_error.

Dataset contents

—polygons/*.parquet: the public polygon split, one shard per source PBF.
—analysis/*.parquet: detailed overlap, provenance, hostname, duplicate, conflict, and per-source statistics.
—deduplication_summary.json: counts and tag-conflict totals from the global canonicalization pass.
—manifests/: source inventory, upload checkpoints, and completion receipt.

Public polygon schema

Column	Type	Nullable	Description
`polygon_id`	`string`	no	Deterministic source-scoped identifier of the form ``<source-stem>:<osm_type>/<osm_id>``.
`region`	`string`	no	Human-readable region label derived from the source PBF filename (Geofabrik convention). Stable and independent of the source filename's extension.
`source_pbf`	`string`	no	Original source PBF filename (no path). Reconstructed from the run's processed-pbfs manifest.
`osm_type`	`string`	no	``"way"` `for a closed polygonal way,` `"relation"`` for an assembled multipolygon or boundary relation.
`osm_id`	`int64`	no	Original OSM numeric identifier of the way or relation. Way and relation namespaces are kept distinct.
`osm_version`	`int32`	no	OSM version number as recorded on the object at extraction time.
`osm_timestamp`	`timestamp[us, tz=UTC]`	no	OSM last-edit timestamp as recorded on the object at extraction time. UTC.
`name`	`string`	yes	Trimmed ``name` `tag value.` `None`` when the tag is absent.
`website`	`string`	yes	Trimmed original ``website` `tag value. Nullable:` `None`` when the tag is absent or whitespace-only.
`contact_website`	`string`	yes	Trimmed original ``contact:website` `tag value. Nullable:` `None`` when the tag is absent or whitespace-only.
`has_website`	`bool`	no	``True` `iff the trimmed` `website`` tag is non-empty.
`has_contact_website`	`bool`	no	``True` `iff the trimmed` `contact:website` `tag is non-empty. Only the exact key is checked; unrelated` `contact:*`` keys do not contribute.
`has_any_website`	`bool`	no	``True` `iff` `has_website` `OR` `has_contact_website. Always` `True`` in the public shard (inclusion invariant).
`website_class`	`string`	yes	Discrete classification of the ``website` `value, or` `None`` when the tag is absent.
`contact_website_class`	`string`	yes	Discrete classification of the ``contact:website` `value, or` `None`` when the tag is absent.
`website_hostname`	`string`	yes	Lowercased hostname extracted from ``website, or` `None`` when the value is not parseable as a URL or bare hostname.
`contact_website_hostname`	`string`	yes	Lowercased hostname extracted from ``contact_website, or` `None`` when the value is not parseable.
`tags`	`string`	no	Deterministic JSON object containing every tag of the source OSM object, with keys sorted. ``{}`` when the object carries no tags.
`tag_keys`	`string`	no	Deterministic JSON array of every tag key of the source OSM object, sorted lexicographically.
`tag_count`	`int32`	no	Number of tags carried by the source OSM object. Always non-negative.
`osm_primary_tag`	`string`	no	Primary OSM category key for the object, selected from a frozen precedence list (``boundary` `>` `building` `>` `amenity` `> ...).` `other`` when no recognised key is present.
`geometry`	`string`	no	Deterministic GeoJSON ``Polygon` `or` `MultiPolygon` `in WGS84. Coordinate order is` `[lon, lat]``. Coordinates are rounded to seven decimal places. Empty geometries are rejected at extraction time and counted as expected exclusions; they never reach this column.
`centroid`	`string`	no	Deterministic GeoJSON ``Point` `in WGS84 ([lon, lat]``) representing the polygon's centroid. Coordinates are rounded to seven decimal places.
`centroid_kind`	`string`	no	Identifier of the centroid algorithm used. Currently always ``"lambert_azimuthal_equal_area"`` -- the centroid is computed in a Lambert azimuthal equal-area projection centred on the polygon's area-weighted outer-ring barycenter and reprojected back to WGS84. NOT a geodesic centroid.
`lat`	`double`	no	Centroid latitude in WGS84 decimal degrees. Rounded to seven decimal places. ``NaN`` and infinity are never produced.
`lon`	`double`	no	Centroid longitude in WGS84 decimal degrees. Rounded to seven decimal places. ``NaN`` and infinity are never produced.
`bbox`	`string`	no	Deterministic JSON array ``[min_lon, min_lat, max_lon, max_lat]`` in WGS84 decimal degrees. Coordinates are rounded to seven decimal places.
`area_m2`	`double`	no	Polygon area in square metres computed on the WGS84 ellipsoid via ``pyproj.Geod``. Outer-ring area minus the absolute area of every inner ring. Always finite and non-negative.
`area_bucket`	`string`	no	Coarse area bucket. One of ``"<10m2",` `"10-100m2",` `"100m2-1km2",` `"1-10km2",` `"10-100km2",` `"100km2-1000km2",` `">=1000km2"``.
`schema_version`	`string`	no	Schema version of the public polygon table. Default extraction and enrichment use ``SCHEMA_VERSION` `(v1.3); the opt-in language stage writes` `v1.4``.
`website_text`	`large_string`	yes	Full main text extracted from ``website`` with Trafilatura; null unless extraction succeeds.
`website_word_count`	`int64`	yes	Number of Python Unicode ``\w+` `sequences in` `website_text``; null without text.
`website_text_status`	`string`	no	Website text enrichment status from the documented frozen vocabulary.
`contact_website_text`	`large_string`	yes	Full main text extracted independently from ``contact:website`` with Trafilatura; null unless extraction succeeds.
`contact_website_word_count`	`int64`	yes	Number of Python Unicode ``\w+` `sequences in` `contact_website_text``; null without text.
`contact_website_text_status`	`string`	no	Contact website text enrichment status from the documented frozen vocabulary.
`website_language`	`string`	yes	Exact script-aware GlotLID language label for ``website_text``; null unless text was successfully extracted and detected.
`website_language_probability`	`double`	yes	Top-1 GlotLID probability for ``website_language``; null without a successful language detection.
`contact_website_language`	`string`	yes	Exact script-aware GlotLID language label for ``contact_website_text``; null unless text was successfully extracted and detected.
`contact_website_language_probability`	`double`	yes	Top-1 GlotLID probability for ``contact_website_language``; null without a successful language detection.
`website_sentences`	`list<element: string not null>`	yes	Sentences segmented from ``website_text` `with SaT; null unless` `website_sentence_status` `is` `success. Segments keep their own trailing whitespace but not the line breaks that separated them, so joining them does not reproduce` `website_text``.
`website_sentence_count`	`int32`	yes	Number of sentences in ``website_sentences``; null unless the text was segmented.
`website_sentence_status`	`string`	no	Why ``website_text` `was or was not segmented:` `success,` `absent` `when no text was extracted,` `unsupported_language` `when the detected language is outside the segmenter's range, or` `empty_text`` when the text carries no segmentable content.
`contact_website_sentences`	`list<element: string not null>`	yes	Sentences segmented from ``contact_website_text` `with SaT; null unless` `contact_website_sentence_status` `is` `success. Segments keep their own trailing whitespace but not the line breaks that separated them, so joining them does not reproduce` `contact_website_text``.
`contact_website_sentence_count`	`int32`	yes	Number of sentences in ``contact_website_sentences``; null unless the text was segmented.
`contact_website_sentence_status`	`string`	no	Why ``contact_website_text` `was or was not segmented:` `success,` `absent` `when no text was extracted,` `unsupported_language` `when the detected language is outside the segmenter's range, or` `empty_text`` when the text carries no segmentable content.

Provenance and license

Source filename, byte size, and nanosecond modification time are recorded before processing. The completion receipt binds finalized artifacts by relative path, byte size, and SHA-256.

The map backdrop uses Natural Earth 1:110m Admin-0 country geography, distributed in the source tree under its public-domain terms.

© OpenStreetMap contributors. OpenStreetMap data is available under the Open Database License (ODbL) 1.0; see the OpenStreetMap copyright and attribution page. Regional PBF extracts are provided by Geofabrik.

Website text is third-party content, separate from the OSM data, and is not covered by the ODbL. This dataset asserts no license for that text and grants no additional reuse rights: copyright and licensing conditions remain with each source website. Check the source site's terms or license before using or redistributing extracted text.

Citation

If you use this dataset, please cite it using the machine-readable metadata in `CITATION.cff`. GitHub and the Hugging Face dataset page can then display the citation directly.

Flandre, Noé. OSM Polygon Website Tag Dataset. Hugging Face dataset