Wanderer210w/vgreen-charging-siting-data

vgreen-charging-siting-data

Data snapshot for the VGreen EV charging-station siting project (Vietnam). Sources: evcs.vn (supply + occupancy telemetry), VinFast official store locator (verification), OSM (roads + POI), WorldPop (population), ESA WorldCover (land use), VNSDI (commune boundaries + registered population). See the project docs/ for the full data-layer overview.

Snapshot: 2026-07-31 · frozen input snapshot id 2026-07-20 (re-frozen 2026-07-31T12:29+07:00, 6 sources — see raw/MANIFEST.json for per-source checksums and freeze provenance; dated raw/MANIFEST-*.json files are the earlier freezes kept for lineage).

Layout

• —raw/ — immutable crawled/downloaded sources (+ MANIFEST.json freeze manifest)
• —interim/ — cleaned / intermediate artifacts + QA sidecars
• —canonical/ — station & connector master tables (Hive-partitioned by province_code)
• —admin/ — commune/province enrichment for stations and the demand grid
• —vnsdi/ — commune polygons + registered population
• —worldpop/ — population on H3 r8, all vintages and repair stages
• —osm/ — roads, POI, Vietnam boundary
• —landuse/ — buildability mask and exclusion zones
• —demand/ — the assembled demand grid
• —processed/ — model-ready outputs (candidate_sites, covered0)
• —external/ — external references (electricity OpEx tariff)

What changed in this version

This is a full national rebuild. Every layer was regenerated, and three new source layers were added. The two caveats that shipped with the 2026-07-28 release (Hanoi-only scope in processed/, and processed/ predating the coordinate fix) are both resolved — see Resolved from the previous release.

New: administrative layer (E-DQ3)

Stations and demand cells now carry commune/province labels from VNSDI 34DVHC layer 2 (Địa phận cấp xã 1000N, admin vintage 2025-06-16) — the post-merger 34-province structure, not OSM adm4.

• —interim/vnsdi/communes.parquet — 3,321 communes, 34 provinces, Σ registered population 113,625,653 (2025 register), Σ area 347,574 km²
• —interim/admin/cell_commune.parquet — 395,276 cell↔commune pairs with area weights, for 314,934 grid cells (73,882 cells straddle more than one commune)
• —interim/admin/demand_commune.parquet — demand aggregated to commune level
• —interim/admin/province_crosswalk.csv — evcs province_code → VNSDI province

New station columns: admin_l1_code, province_name, commune_code, commune_name, commune_kind, admin_src, admin_dist_m, admin_verdict.

Two things to keep straight:

• —Cell labels are `argmax`, not allocation. A cell's commune_code is the commune holding the largest share of it. To split a cell's population across communes, join cell_commune.parquet and use the area weights (they sum to 1 per cell, max drift 2.2e-16).
• —326 cells (1,471 people, 0.0015%) fall outside every commune polygon and are labelled UNRESOLVED rather than dropped.

New: population repair chain (E-DQ7e / E-DQ7f / E-DQ8b)

The population raster moved to the UN-adjusted 2020 constrained build, then through two repair stages. Each stage ships its own parquet and report so you can pick the stage you trust:

• —E-DQ7f cross-checks WorldPop cells against VNSDI commune DANSO. 139 cells holding 745,283 people (0.76%) were implausible; 70 communes were repaired (17 re-totalled, 53 replaced), removing 486,399 phantom people. National drift 0.499%, 0 cells left unrepaired.
• —E-DQ8b relocates population sitting in cells with no road access. Roadless mass drops from 1,183,197 to 1,001 people (−99.9%); all relocated mass lands in DIRECT cells. Cells carry an access_tier: DIRECT 98.7%, ADJACENT 0.92%, NEAR 0.16%, ISOLATED 0.19%.

Two extra vintages ship for per-cell comparison work: worldpop_pop_2025_h3.parquet (Σ 101,299,971, R2024B) and worldpop_pop_2020_r24_h3.parquet (same R2024B generation as 2025, so the pair is generation-matched).

Note. VNSDI's registered population is 16.5% above WorldPop's UN-adjusted total (113.6 M vs 97.6 M). These count different things — de jure register vs modelled de facto — so the gap is expected and E-DQ7f uses the register only to detect outlier cells, not to rescale the country.

New: POI semantics and boundary clipping (E-DQ7a / E-DQ7b)

• —osm/vn_boundary.parquet / .geojson — Vietnam boundary; the demand grid is now clipped to it. 7,375 cells fall outside and are split out into demand/demand_h3_clipped_out.parquet (6,338 people, 0.0065% — negligible).
• —osm/osm_poi_h3.parquet (6,932 cells) — POI counts kept separate from road access, which previously conflated the two signals.
• —osm/osm_poi_recall.json — measured POI recall against the station set, and it is low: FUEL 0.359 at 100 m, PARKING_OFF 0.086. Recall is also biased by population density (PARKING_OFF high/low-density ratio 2.67). Treat OSM POI counts as a weak, unevenly-observed covariate — not a census.

New: asset vs live capacity resolution (E-DQ4)

evcs.vn reports currently reporting guns, which undercounts installed hardware. The canonical table now separates the two, with VinFast official as the asset source of truth (config_src: OFFICIAL 19,243 / UNKNOWN 256 / EVCS_LIVE 7 / TELEMETRY_BOUND 1):

This recovers 6,249 guns that live telemetry missed. Do not sum `nameplate_power_kw` as site capacity — 4,317 stations share cabinets between guns, and nameplate over-states deliverable site power by 1.82×; use site_power_kw.

Canonical stations — 19,507 rows, 56 columns

Identity columns (E-DQ2, cross-source dedup between evcs and VinFast official) and coordinate-provenance columns (E-DQ1) carry forward from the previous release: physical_id, is_primary, dup_group_id, dup_dist_m, n_dup_members, lat_raw, lng_raw, coord_src, coord_resolved, coord_fix_dist_m.

Two changes since 2026-07-28:

• —Dedup was re-run against the rebuilt official cross-reference; is_primary=False is now 146 rows (was 329).
• —coord_resolved=False is now 54 rows (was 38) — the 38 placeholder coordinates plus 16 newly detected as falling outside every commune polygon.

Rows are flagged, never deleted. The supply filter is:

supply = df["is_operational"] & (df["access"] == "PUBLIC") & df["is_primary"] & df["coord_resolved"]
# -> 19,181 stations across 12,801 H3 r8 cells

Pre-computed as interim/clean_supply.csv (19,181 rows), with every excluded row and its reason in interim/excluded.csv (326 rows). Reasons are assigned in precedence order, so these are disjoint:

h3_r8 is NULL on unresolved rows — those stations have no trustworthy location and must not anchor coverage.

`MAINTENANCE` is in supply on purpose. 3,392 stations are MAINTENANCE rather than OPERATIONAL; they are physically installed, so they count as existing supply. Only OUT_OF_SERVICE is hard-filtered. If you want the stricter "operational right now" baseline, use processed/covered0_operational.

Private ownership is not private access. Operator values like Tư nhân / Nhượng quyền describe ownership or franchise model, not public accessibility. They are the majority of supply and belong in it. Use the access column, not the operator string.

Demand grid — 314,934 cells

interim/demand/demand_h3.parquet, H3 r8, national. Σ pop_adj 96,941,979, Σ pop_2025 101,299,971, 578,473 km of road length. Carries the access_tier, admin labels, road metrics, and POI counts described above. All 9 QA gates PASS (demand/demand_h3_report.json).

processed/ — national, rebuilt against this canonical layer

covered0 applies the supply filter plus an F4 clean_coord check, which is why it holds 19,012 rows rather than 19,181 — a net 169 fewer. (The dirty_coord: 214 figure in covered0_report.json counts every row failing that check, including rows the supply filter already excluded for another reason, so the per-reason counts there overlap and do not sum to the total.)

Resolved from the previous release

⚠️ Known issues in this release

1. Three QA sidecars are stale. They were generated on 2026-07-30, before the final rebuild, and their headline counts no longer match the shipped data:

interim/export_supply_report.json (2026-07-31) is the authoritative supply report. The gate results in the stale files still hold — it is the row counts that drifted. Regenerating them is tracked as follow-up.

2. `interim/canonical/stations_no_connectors.csv` was removed. It was redundant with the connector table; derive it with an anti-join if you need it.

3. `raw/vinfast_official/details.tar.zst` and `interim/evcs_timeseries.tar.zst` are unchanged from the 2026-07-24 bundle. They are still consistent with the frozen 2026-07-20 snapshot.

Bundled archives (extract before use)

Two folders that contained tens of thousands of tiny files were bundled into single compressed archives to keep the repo pushable within Hugging Face rate limits. Extract them to restore the original per-file layout:

# 23,240 per-locator detail JSONs  ->  raw/vinfast_official/details/
zstd -dc raw/vinfast_official/details.tar.zst | tar -x -C raw/vinfast_official

# 19,218 per-station time-series CSVs  ->  interim/evcs_timeseries/
zstd -dc interim/evcs_timeseries.tar.zst | tar -x -C interim

Reading the canonical layer

Both canonical tables are Hive-partitioned by province_code, and filenames are content-hash prefixed — a rebuild writes new filenames, so read the directory, never a pinned file path:

import pyarrow.dataset as ds

stations = ds.dataset("interim/canonical/stations",
                      format="parquet", partitioning="hive").to_table().to_pandas()