lebfla11/europe-holdout-2k

Europe-Holdout-2k

A 2,000-location street-level geolocation benchmark sampled from the held-out test fold of the thesis training pipeline. Designed as a high-variation, contamination-free probe for the published thesis models (v14 unfreeze_last2, v15, ...) and any external geolocation system.

Why a holdout-derived benchmark?

The thesis pipeline scrapes ~79k locations from Google Street View across 112 countries. All training, validation and reported numbers use a single deterministic 80/10/10 split (seed=42, location-level). The 10% test fold (~7,750 EU locations) is never touched during training.

Sampling 2k locations from this fold gives:

1. 1.Zero contamination for any thesis model — these locations were not used to fit weights, calibration, or evaluation hyperparameters.
2. 2.Direct comparability across the unified leaderboard — the same 2k rows evaluate v14 unfreeze_last2, v15, the StreetCLIP baselines, etc.
3. 3.Cross-domain stability — a complementary view to osm_europe_1k, which uses Mapillary/OSV-5M imagery from a different distribution.

Stratification

Country-stratified to keep small countries from disappearing under proportional sampling. Allocation rule per country:

alloc[c] = max(min(floor=5, holdout_count[c]), round(2000 * holdout_count[c] / total))

Excess/shortage after rounding is reconciled by adding/removing single units from the strata with the largest residuals (largest residuals get extras; largest over-allocations get trimmed). The total is exactly 2000.

42 countries are represented; the largest five (FR, IT, DE, ES, GB) account for ~60% of rows, mirroring the underlying scraped distribution.

Files

• —metadata.csv — filename,lat,lon,country,osv5m_id (osv5m_id is empty; these points are not from OSV-5M)
• —sample.py — re-build the manifest from geocell_map.json + all_locations.csv (deterministic; same output every run)
• —download_gsv.py — fetch Google Street View images at the listed coordinates using the user's own API key. 4 headings per row (0/90/180/270). Writes images_gsv/ and metadata_paired.csv
• —eval.py — standalone evaluator (haversine error + accuracy at 1/25/200/750/2500 km). No project imports

Reproducing the manifest

python sample.py \
    --geocell_map ../../data_collection/outputs/geocells_k4000_eu/geocell_map.json \
    --locations  ../../data_collection/outputs/analysis/all_locations.csv \
    --n 2000 --seed 42 --floor 5

Both the split RNG and the sampling RNG are seeded; running twice on the same inputs yields byte-identical metadata.csv.

Fetching imagery

export API_KEY=...        # your Google Maps Static Street View key
export API_SECRET=...     # optional URL-signing secret
python download_gsv.py    # writes images_gsv/00000_0.jpg ... 01999_270.jpg

Cost is bounded by 1 metadata-check + up to 4 image-fetches per row. With the standard $200/month free credit, the full 2k fetch fits comfortably.

Evaluating a model

eval.py reads predictions from a CSV/JSON with at minimum filename, pred_lat, pred_lon:

python eval.py --predictions runs/eval_holdout2k/your_model/predictions.json

License

• —metadata.csv is licensed CC-BY-SA 4.0
• —The Python scripts (sample.py, download_gsv.py, eval.py) are MIT
• —Google Street View image bytes fetched via `download_gsv.py` are NOT redistributable — they are subject to Google's Maps Platform Terms of Service. Download for your own evaluation only