CoolFace
Datasetpublic

NoeFlandre/osm-polygon-website-tag

OSM Polygon Website Dataset OpenStreetMap polygons that carry a website or contact:website tag, with the full main-page text of each site. Every number below is recomputed from the published Parquet files. At a glance Polygons 1,726,474 With extracted text 1,192,980 Words of text 407,685,655 Languages 397 Regional sources 386 / 386 Duplicate objects removed 104,927 Candidates rejected 868,905,743 Status In progress… See the full description on the dataset page: https://huggingface.co/datasets/NoeFlandre/osm-polygon-website-tag.

sourceHugging Faceodblupdated 2d agoView on Hugging Face
2likes2.4kdownloads
Dataset Card

OSM Polygon Website Dataset

[image]

OpenStreetMap polygons that carry a website or contact:website tag, with the full main-page text of each site. Every number below is recomputed from the published Parquet files.

At a glance

Polygons1,726,474
With extracted text1,192,980
Words of text407,685,655
Languages397
Regional sources386 / 386
Duplicate objects removed104,927
Candidates rejected868,905,743
StatusIn progress

Website text

TagURLsSuccessfulEmptyFailedWords
website1,518,6051,075,91056,248386,447368,299,585
contact:website169,238123,5204,93540,78339,386,070

Counts are unique (osm_type, osm_id) polygons -- 407,685,655 words in total. Regional overlap duplicates are removed globally.

Languages

Detected with GlotLID v3. Labels are script-aware language_Script codes; each row carries its top-1 probability.

MetricValue
Distinct languages397
Labeled website texts1,075,910
Labeled contact:website texts123,520

Top 10 labels across both tags:

LanguageTexts
eng_Latn533,896
deu_Latn216,762
fra_Latn85,516
pol_Latn42,731
nld_Latn30,533
spa_Latn30,358
rus_Cyrl28,691
ita_Latn25,038
jpn_Jpan24,208
por_Latn22,377

Sentences

Segmented with SaT, which covers 85 languages; anything else records unsupported_language. Segments do not rejoin into the source text -- use the *_text columns for that.

MetricValue
Sentences45,823,447
Segmented website texts1,073,468
Segmented contact:website texts125,132
Eligible text units for splitting1,236,073
Split with a supported language1,198,600
Left unsplit: unsupported language37,473
Sentence-splitting coverage97.0%
Unsupported-language share3.0%

Mean sentences per segmented text: 38.2

Most common unsupported languages: kiu_Latn (12,502), hrv_Latn (4,715), zxx_Zzzz (4,031), anp_Deva (3,014), bos_Latn (1,902).

Geographic distribution

[image]

5,850 occupied H3 cells at resolution 3, covering 1,192,980 unique polygons with extracted text. Log colour scale, Natural Earth 1:110m backdrop.

Polygon geometry

Geodesic areas on the WGS84 ellipsoid, over every published polygon row. Full breakdown in `stats.json`.

MetricValue
Polygons measured1,726,474
Total area906,230.67 km²
Median area946.58 m²
Mean area524,902.59 m²
Smallest / largest area0.01 / 129,636,243,089.97 m²
p95 area82,053.29 m²
MultiPolygon rows32
Rows with holes20
Rows below 1 m²393

Dataset bounding box: [-179.992396, -77.847199, 179.325994, 82.172551] (min lon, min lat, max lon, max lat).

Top website hostnames

HostnamePolygons
www.facebook.com10,238
www.mcdonalds.com9,886
tools.usps.com6,382
www.dollargeneral.com5,909
locations.tacobell.com4,579

Top contact:website hostnames

HostnamePolygons
www.7-eleven.com1,587
www.tesco.com1,441
5ka.ru1,224
rosneft-azs.ru848
www.mcdonalds.com827

Dataset contents

  • —polygons/*.parquet -- the polygons and their extracted text, one shard per source.
  • —analysis/*.parquet -- languages, sentences, hostnames, duplicates and per-source counts.
  • —stats.json, deduplication_summary.json -- full geometry and dedup numbers.
  • —manifests/ -- source inventory and completion receipt.

Public polygon schema

ColumnTypeNullable
polygon_idstringno
regionstringno
source_pbfstringno
osm_typestringno
osm_idint64no
osm_versionint32no
osm_timestamptimestamp[us, tz=UTC]no
namestringyes
websitestringyes
contact_websitestringyes
has_websiteboolno
has_contact_websiteboolno
has_any_websiteboolno
website_classstringyes
contact_website_classstringyes
website_hostnamestringyes
contact_website_hostnamestringyes
tagsstringno
tag_keysstringno
tag_countint32no
osm_primary_tagstringno
geometrystringno
centroidstringno
centroid_kindstringno
latdoubleno
londoubleno
bboxstringno
area_m2doubleno
area_bucketstringno
schema_versionstringno
website_textlarge_stringyes
website_word_countint64yes
website_text_statusstringno
contact_website_textlarge_stringyes
contact_website_word_countint64yes
contact_website_text_statusstringno
website_languagestringyes
website_language_probabilitydoubleyes
contact_website_languagestringyes
contact_website_language_probabilitydoubleyes
website_sentenceslist<element: string not null>yes
website_sentence_countint32yes
website_sentence_statusstringno
contact_website_sentenceslist<element: string not null>yes
contact_website_sentence_countint32yes
contact_website_sentence_statusstringno

Method

  • —Geometry assembled with libosmium; text extracted with Trafilatura and never truncated. Word counts are Unicode \w+ matches.
  • —Text status is one of absent, pending, success, empty, invalid_url, unsafe_url, fetch_error, extract_error. A source is enriched only when every status is success or absent. Failed values retry on later resumptions; successful values are cached.
  • —URLs resolving to anything other than a public IP are refused as unsafe_url, before and after redirects.
  • —Live metrics · source code

Provenance and license

Every artifact is bound by relative path, byte size and SHA-256 in the completion receipt. Map backdrop: Natural Earth 1:110m (public domain).

© OpenStreetMap contributors, under the ODbL 1.0 -- see the copyright page. Extracts from Geofabrik.

The website text is not covered by the ODbL. It is third-party content; rights stay with each source site. Check a site's terms before reusing its text.

Citation

Machine-readable metadata: `CITATION.cff`.

Flandre, Noé. OSM Polygon Website Tag Dataset. Hugging Face dataset