KokosDev/mur-sst-ml-benchmark-Zarr

MUR SST ML Benchmark (Pacific, Zarr)

Machine-learning friendly Zarr subset of NASA/JPL GHRSST MUR SST.

• —Upstream source (public, no auth): s3://mur-sst/zarr
• —Subset:
• —Region: Pacific (20–50°N, 180–240°E); longitude is stored as 0–360°E
• —Time: 2018-01-01 → 2019-12-30 (729 daily frames; upstream coverage for this slice ends on 2019-12-30)
• —Variable: analysed_sst only (float32, °C)
• —Chunking for ML: (time, lat, lon) = (7, 256, 256) (weekly windows)

Why mur-sst/zarr-v1 during extraction?

• —mur-sst/zarr is chunked with the entire time axis in one chunk, making time subsetting extremely inefficient.
• —mur-sst/zarr-v1 is time-chunked and enables practical extraction. The output dataset here is the requested ML rechunk.

Notes (Hub viewer)

• —The Dataset Viewer is expected to be unavailable because this repo contains a tar archive of a Zarr store (not a datasets-native format with named splits).

Files in this dataset repo

Because Hugging Face dataset repos + Git LFS handle a single large file much more reliably than tens of thousands of tiny chunk files, the Zarr store is published as:

• —pacific_sst.zarr.tar (a tar archive of the pacific_sst.zarr/ directory)

To use it locally:

tar -xf pacific_sst.zarr.tar

SST forecasting task definition

We define a next-week forecasting task:

• —Input: 7 daily SST frames, shape (7, H, W)
• —Target: next 7 daily SST frames, shape (7, H, W)
• —Goal: learn a function that predicts the next week from the previous week

Windows are created from the time axis; you can use overlapping or non-overlapping windows (benchmark scripts default to non-overlapping).

Train/val/test splits

Time-contiguous splits (no leakage):

• —Train: 2018-01-01 → 2018-12-30
• —Val: 2018-12-31 → 2019-06-30
• —Test: 2019-07-01 → 2019-12-30

Streaming code example

Local:

import xarray as xr
ds = xr.open_zarr("pacific_sst.zarr", consolidated=True)
print(ds)

Remote (Hugging Face, after download):

import xarray as xr

# 1) Download pacific_sst.zarr.tar from the Hub
# 2) tar -xf pacific_sst.zarr.tar
ds = xr.open_zarr("pacific_sst.zarr", consolidated=True)
print(ds)

Benchmark results

Run:

tar -xf pacific_sst.zarr.tar
python bench/throughput_benchmark.py --local pacific_sst.zarr --s3-root mur-sst/zarr-v1

Measured on this machine (see bench/throughput_benchmark.py for details):