dcher95/PPE-Global

PPE-Global

Version 1.1 — changes from v1.0 (GIFT woodiness re-fetched against every species; observer-disjoint split_observer added; every other table identical) are itemised in PROVENANCE.md §6–7.

A global plant-phenology dataset built from iNaturalist research-grade observations. Metadata only — imagery is referenced by photo_id and fetched from the public inaturalist-open-data S3 bucket (see Images below).

Extends its predecessor Pheno3M/PPE (3,573,640 observations, 6,825 species, US-only) to 63,098,434 observations across 154,676 species and 245 countries.

Contents

Photos per observation: 1.82. 26.4% of photos appear in TreeOfLife-200M (BioCLIP-2's training set) — flagged per photo as in_tol, so contamination can be filtered exactly rather than approximated by a year cutoff.

Documentation

Bundled in this repo — read them in this order:

from huggingface_hub import snapshot_download
root = snapshot_download("dcher95/PPE-Global", repo_type="dataset")

Two phenology axes — and check label_source first

The mask columns are 4-bit multi-hot fields, and every one of them names who produced it:

Provenance is spelled three ways so you cannot miss it:

• —`label_source` — a string: "human", "machine", "both", "none". Read this one.
• —`has_human` / `has_machine` — the booleans to filter on.
• —`label_method` — the same thing packed into 2 bits (bit 0 human, bit 1 machine).

pairs_annotated.parquet carries label_method_a/_b, both_human, and its own label_source. The reference loader takes human_labels_only=True.

Train on everything; score only where `has_human`. L_flow-style objectives regress between two embeddings and consume no phase label, so provenance decides nothing but which observations enter the pool. Anything you report is different.

Detectors cannot express two of the four classes. They emit flower and fruit presence and nothing else, so Budding — the class budburst models exist to predict — and every true absence in this corpus come only from human annotation.

The vegetative axis matters for a separate reason: a reproductive label alone conflates a bare February twig with a leafy July shrub — both are "no flowers or fruits". Darwin Core has no vegetative-phenology term, so this axis is absent from GBIF-derived datasets entirely.

Pairs are sampled, not stored

There are ~1.4 billion co-located same-species pairs within 1 km (~57 GB), so they are not materialized. Each observation carries spatial cell keys; a pair is drawn by picking a cluster from cluster_index.parquet and then two members. Weighting clusters by n(n-1)/2 reproduces uniform-over-pairs exactly; weighting by 1 maximizes site diversity. A tier is therefore a sampler predicate, not a fixed subset.

Reference loader: ppe_global_loader.py.

Images

Not redistributed. Rebuild from photos.parquet:

https://inaturalist-open-data.s3.amazonaws.com/photos/<photo_id>/medium.<extension>

medium (~142 KB) is the right rendition for 224 px encoders. One photo per observation is ~8.2 TB at full corpus; the tight-radius tiers are ~157 GB.

Caveats — please read before using

• —Co-located, not same-individual. Pairs are same species, same place, different time. Manual verification was attempted and failed: 28 expert labels returned 64% undecidable, and controls known to be different plants were correctly rejected only 3 times in 7. iNaturalist photos are close-ups without the scene context needed to match a location. No individual-level claim is supported.
• —Most of the reproductive axis is machine-generated — check `label_method` before scoring. The reproductive labels come from two places and they are not interchangeable:

repro_mask is the union of the first two, provided for training-pool selection only. label_method is a 2-bit flag — bit 0 "has a human label", bit 1 "has a machine label" — and pairs_annotated.parquet carries label_method_a/_b plus the derived both_human. Train on whatever you like; evaluate only where bit 0 is set.

• —`phenovision.parquet` is an older checkpoint of the same detector that Phenobase republishes (10.57967/hf/2763, through March 2024, vs hf/7952). Its rows largely overlap repro_mask_machine rather than adding to it — do not sum the two. The overlap is computed rather than assumed: each row carries dup_of_phenobase, and only the rows where it is false are new coverage. Like all machine labels it is presence-only: absences and equivocal predictions are not published upstream, so "no flower" is indistinguishable from "not scored", and it has no flower-bud class.
• —Machine labels cannot express two of the four classes. Detectors emit flower and fruit presence, nothing else. Budding and every true absence in this corpus exist only because of the human term-12 pull, which is why that pull is small but disproportionately valuable.
• —Sampling is biased. Effort concentrates in urban and developed areas, on conspicuous species, and in recent years. Observers are heavy-tailed.
• —Coordinate precision varies. coord_unc_m is user/device reported and absent on ~23% of records. iNaturalist obscures coordinates for sensitive taxa; open data carries no flag for this, so large coord_unc_m is the only proxy.
• —`rank_ok` marks identifications at species level or finer. Genus-level records cannot be paired by species.
• —Cell recall is ~99.4–99.98%, not exact: candidate generation uses a 3×3 block of metre-quantized cells and the residual loss is pairs straddling a 1° latitude band.

Provenance

Built by src/ppe_global/stages/ from the iNaturalist Open Data export (CC0/CC-BY/CC-BY-NC subset), Phenobase (machine flower/fruit labels), the iNaturalist API (human leaf and term-12 annotations), GIFT (woodiness) and PhenoVision (machine labels, CC-BY-4.0, Zenodo 15306421). Every number above is read from the pipeline's own stage receipts. How each label was produced, and why the pair count is what it is, is written up in PROVENANCE.md.