calebrob6/similar-but-different

Similar But Different — Sentinel-2 patches with deceptive RGB

30,927 32×32 Sentinel-2 L2A multispectral patches (12 bands resampled to 10 m) across ten ESA WorldCover classes, selected so that the visible bands are uninformative by construction: every patch sits in a region of RGB-mean space dominated by patches of other classes, while its NIR / red-edge / SWIR response stays class-informative.

The dataset is a controlled probe for one question: does a model actually use the bands outside the visible range? On most land-cover benchmarks an RGB-only model nearly matches a full multispectral model, so band ablations have no leverage. Here an RGB-only ResNet-18 reaches 0.71 test accuracy while the same model with all 12 bands reaches 0.82 — and frozen multispectral foundation-model probes reach 0.87. See the announcement post at <https://geospatialml.com/posts/similar-but-different/> for full benchmarks.

How it was built

Imagery is Sentinel-2 L2A from the Microsoft Planetary Computer; labels come from ESA WorldCover 10 m 2021 v200.

1. 1.Candidate sampling. Random 32×32 windows from Sentinel-2 L2A scenes (scene-level cloud cover < 10%) at 200 hand-picked locations spread across continents, biomes, and seasons; windows containing cloud or shadow pixels (per the SCL band) are discarded. This yields 397,839 patches from 5,292 scenes, with per-channel RGB means and standard deviations computed on the 8-bit visual (true-colour) asset.
2. 2.Spectral-difference filtering. Find all patch pairs whose RGB-mean vectors are within Euclidean distance 2.0 and whose RGB-std vectors are within distance 3.0 (so pairs match in texture, not just average colour). For those candidates, fetch the NIR band (B08) and keep pairs whose B08 means differ by a large margin — every surviving patch has at least one RGB look-alike with a very different NIR response.
3. 3.Full bands + labels. Download all 12 L2A surface-reflectance bands for surviving patches (20 m / 60 m bands resampled to 10 m, bilinear) and label each patch with the mode WorldCover class over its footprint, keeping the full per-class percentage distribution.
4. 4.RGB-ambiguity selection. Restrict to patches ≥ 70% a single WorldCover class that participate in a pair with |ΔB08| > 1000 (uint16 surface reflectance), then compute each patch's k = 20 nearest neighbours in (R̄, Ḡ, B̄) space and keep the patch only if ≥ 18 of its 20 RGB-mean neighbours carry a different label. A k-NN / nearest-centroid classifier on RGB means is therefore wrong on these patches by construction. Snow and ice survived with too few patches to split and was dropped, leaving ten classes and 30,927 patches.

What's in this repo

similar-but-different/
├── README.md                       # this file
├── splits.json                     # scene-disjoint 80/10/10 train/val/test
├── landcover_distribution.parquet  # per-patch WorldCover percentages
├── eval/
│   ├── test_pairs.parquet          # RGB-similar / NIR-different test pairs
│   └── test_hard_clusters.parquet  # RGB-confusable, multi-class test clusters
└── shards/
    ├── Bare_sparse_veg.parquet     # one parquet shard per class
    ├── Built-up.parquet
    ├── Cropland.parquet
    ├── Grassland.parquet
    ├── Herbaceous_wetland.parquet
    ├── Mangroves.parquet
    ├── Moss_and_lichen.parquet
    ├── Permanent_water.parquet
    ├── Shrubland.parquet
    └── Tree_cover.parquet

Each shard row holds one patch: the complete GeoTIFF as tif bytes plus metadata columns (patch_id, scene_id, wc_mode_class, wc_mode_label, wc_mode_pct, wc_n_valid, and wc_class_<code>_pct for the full WorldCover distribution over the patch footprint).

Each embedded GeoTIFF is a 12-band uint16 raster, 32×32 px at 10 m, georeferenced in the source scene's UTM CRS (nodata=0, compress=deflate). Values are L2A surface reflectance (divide by 10,000 for reflectance in [0, 1]). GeoTIFF tags carry patch_id, scene_id, wc_mode_label, wc_mode_pct, and the patch centre in UTM coordinates; per-band descriptions name each band.

Band order

Non-10 m bands were resampled to 10 m with bilinear interpolation. (B10 cirrus is not part of the L2A product, so it is not included.)

Splits

splits.json provides a fixed 80/10/10 train/val/test split that is stratified by class and grouped by source Sentinel-2 scene (sklearn.model_selection.StratifiedGroupKFold(n_splits=10) with scene_id as the group, seed 42; fold 0 = test, fold 1 = val, folds 2–9 = train). No scene contributes patches to more than one split, so a model cannot match a test patch to a visually near-identical neighbour from the same scene seen during training. Please use this split when reporting results.

The file maps each split to {class_name: [patch_id, ...]} and also stores class_to_idx and inverse-frequency class weights (clipped at 10.0) for weighted cross-entropy.

Evaluation groupings (eval/)

Two extra artifacts under eval/ make the dataset's similar-but-different structure directly testable. Both are built only from the test split, from the patches' visual-RGB statistics, so a model trained on train has never seen them — no need to retrain to use them. They ask a sharper question than plain per-patch accuracy: when two (or more) patches look alike in RGB but carry different labels, does the model tell them apart?

`eval/test_pairs.parquet` — 46 pairs of test patches that are close in visual RGB (mean L2 ≤ 2.0 and per-channel std L2 ≤ 3.0) yet differ by a large near-infrared margin (|ΔB08 mean| > 500 SR units). Columns: patch_a, patch_b, label_a, label_b, rgb_l2, std_l2, nir_mean_diff, re_mean_diff. The similar-but-different pairs are the cross-label ones (label_a != label_b, 70% of rows). Pair accuracy = fraction of pairs where the model classifies both endpoints correctly.

`eval/test_hard_clusters.parquet` — 53 hard clusters (121 patches, long format: one row per membership). Clusters are the connected components of the test-set RGB mean+std similarity graph (same thresholds) that contain ≥ 2 patches spanning ≥ 2 distinct labels — groups that look alike in RGB but cross class boundaries. Columns: cluster_id, patch_id, wc_mode_label, cluster_size, n_labels (sizes: 42×2, 7×3, 4×4). Cluster accuracy = fraction of clusters where the model classifies all members correctly (the size-2 case reduces to pair accuracy).

import pandas as pd
# pred: dict mapping patch_id -> predicted class label (string)

pairs = pd.read_parquet(f"{local}/eval/test_pairs.parquet")
both_correct = (pairs.patch_a.map(pred).eq(pairs.label_a) &
                pairs.patch_b.map(pred).eq(pairs.label_b))
pair_accuracy = both_correct.mean()

clusters = pd.read_parquet(f"{local}/eval/test_hard_clusters.parquet")
ok = clusters.assign(hit=clusters.patch_id.map(pred).eq(clusters.wc_mode_label))
cluster_accuracy = ok.groupby("cluster_id").hit.all().mean()

Reference — a random forest on per-band mean+std (fit on train, the same baseline as below) scores:

RGB-only statistics can barely resolve these groups; the non-visible bands are what make them separable — which is the whole point of the dataset.

Quick start

import io, json
import pandas as pd
import rasterio
from huggingface_hub import snapshot_download

local = snapshot_download(repo_id="calebrob6/similar-but-different",
                          repo_type="dataset")

# Read one patch from a shard:
shard = pd.read_parquet(f"{local}/shards/Tree_cover.parquet")
row = shard.iloc[0]
with rasterio.open(io.BytesIO(row["tif"])) as src:
    arr = src.read()              # (12, 32, 32) uint16 surface reflectance
    print(src.descriptions[7])    # 'B08 NIR 842nm'
    print(src.tags())             # {'patch_id': ..., 'wc_mode_label': ...}

# Use the fixed scene-disjoint split:
splits = json.loads(open(f"{local}/splits.json").read())
train_ids = {pid for ids in splits["train"].values() for pid in ids}
train_rows = shard[shard["patch_id"].isin(train_ids)]

Or with the HF datasets library:

from datasets import load_dataset
ds = load_dataset("calebrob6/similar-but-different",
                  data_files="shards/*.parquet")

Baseline results

ResNet-18 (ImageNet init, input conv adapted to the channel count; AdamW + cosine schedule, 50 epochs, batch 256, flips/rotations; reflectance / 10,000). Best run per band set from a sweep over learning rate {3e-4, 1e-3, 3e-3} × class weighting {none, inverse}, on the fixed split above:

The ordering RGB < RGB+NIR < All holds in every cell of the sweep, and re-running the winning configurations with a different random seed gives 0.717 / 0.772 / 0.817. Frozen foundation-model probes (KNN-5 and linear, in the style of torchgeo-bench) show the same pattern — band-flexible models such as DOFA, Panopticon, and OlmoEarth gain 6–24 points from RGB to the full stack, with an OlmoEarth-v1-Base linear probe reaching 0.867 — see the announcement post for the full table.

Note that the dataset is a controlled probe, not a representative sample of the land surface: class priors and geography reflect the selection procedure, and labels inherit WorldCover's errors (patches are ≥ 70% a single class, but residual label noise remains). Read accuracies as measurements of band reliance, not as operational land-cover accuracy.

License

Released under CC BY 4.0, matching the upstream Sentinel-2 L2A data (Copernicus open access) and ESA WorldCover (CC BY 4.0).

Attribution: Contains modified Copernicus Sentinel-2 L2A data (2024–2025); land-cover labels derived from © ESA WorldCover 2021 v200.

Citation

@misc{similar-but-different-2026,
  title  = {Similar But Different: a Sentinel-2 benchmark for probing
            spectral-band reliance in land-cover models},
  author = {Robinson, Caleb},
  year   = {2026},
  url    = {https://huggingface.co/datasets/calebrob6/similar-but-different}
}