Datasetpublic

LEAP/ChaosBench

ChaosBench: A Multi-Channel, Physics-Based Benchmark for Subseasonal-to-Seasonal Climate Prediction NeurIPS 2024 Oral ChaosBench is a benchmark project to improve and extend the predictability range of deep weather emulators to the subseasonal-to-seasonal (S2S) range. Predictability at this scale is more challenging due to its: (1) double sensitivities to intial condition (in weather-scale) and boundary condition (in climate-scale), (2) butterfly effect, and… See the full description on the dataset page: https://huggingface.co/datasets/LEAP/ChaosBench.

sourceHugging Facegpl-3.0updated 1y agoView on Hugging Face

7likes5.8kdownloads

Dataset Card

ChaosBench: A Multi-Channel, Physics-Based Benchmark for Subseasonal-to-Seasonal Climate Prediction

<div align="center"> <h3>NeurIPS 2024 Oral</h3> </div>

ChaosBench is a benchmark project to improve and extend the predictability range of deep weather emulators to the subseasonal-to-seasonal (S2S) range. Predictability at this scale is more challenging due to its: (1) _double sensitivities to intial condition (in weather-scale) and boundary condition (in climate-scale), (2) butterfly effect, and our (3) inherent lack of understanding of physical processes operating at this scale. Thus, given the high socioeconomic stakes_ for accurate, reliable, and stable S2S forecasts (e.g., for disaster/extremes preparedness), this benchmark is timely for DL-accelerated solutions.

🪜 Leaderboard

View our interactive leaderboard dashboard here

You can access the full score and checkpoints in logs/<MODEL_NAME> within the following subdirectory:

—Scores: eval/<METRIC>.csv
—Model checkpoints: lightning_logs/

✨ Features

1️⃣ _Diverse Observations_. Spanning over 45 years (1979-), we include ERA5/LRA5/ORAS5 reanalysis for a fully-coupled Earth system emulation (atmosphere-terrestrial-sea-ice)

2️⃣ _Diverse Baselines_. Wide selection of physics-based forecasts from leading national agencies in Europe, the UK, America, and Asia

3️⃣ _Differentiable Physics Metrics_. Introduces two differentiable physics-based metrics to minimize the decay of power spectra at long forecasting horizon (blurriness)

4️⃣ _Large-Scale Benchmarking_. Systematic evaluation (deterministic, probabilistic, physics-based) for state-of-the-art ML-based weather emulators like ViT/ClimaX, PanguWeather, GraphCast, and FourcastNetV2

🏁 Getting Started

_NOTE:_ Only need the dataset? Jump directly to Step 2. If you find any problems, feel free to contact us or raise a GitHub issue.

Step 0: Clone the ChaosBench Github repository

Step 1: Install package dependencies

$ cd ChaosBench
$ pip install -r requirements.txt

Step 2: Initialize the data space by running

$ cd data/
$ wget https://huggingface.co/datasets/LEAP/ChaosBench/resolve/main/process.sh
$ chmod +x process.sh

Step 3: Download the data

# Required for inputs and climatology (e.g., for normalization; 1979-)
$ ./process.sh era5
$ ./process.sh lra5
$ ./process.sh oras5
$ ./process.sh climatology

# Optional: control (deterministic) forecasts (2018-)
$ ./process.sh ukmo
$ ./process.sh ncep
$ ./process.sh cma
$ ./process.sh ecmwf

# Optional: perturbed (ensemble) forecasts (2022-)
$ ./process.sh ukmo_ensemble
$ ./process.sh ncep_ensemble
$ ./process.sh cma_ensemble
$ ./process.sh ecmwf_ensemble

# Optional: state-of-the-art (deterministic) forecasts (2022-)
$ ./process.sh panguweather
$ ./process.sh fourcastnetv2
$ ./process.sh graphcast

🔍 Dataset Overview

All data has daily and 1.5-degree resolution.

1._ERA5 Reanalysis for Surface-Atmosphere (1979-2023). The following table indicates the 48 variables (channels) that are available for Physics-based models. Note that the Input ERA5 observations contains ALL_ fields, including the unchecked boxes:

Parameters/Levels (hPa)	1000	925	850	700	500	300	200	100	50	10
Geopotential height, z ($gpm$)	&check;	&check;	&check;	&check;	&check;	&check;	&check;	&check;	&check;	&check;
Specific humidity, q ($kg kg^{-1}$)	&check;	&check;	&check;	&check;	&check;	&check;	&check;
Temperature, t ($K$)	&check;	&check;	&check;	&check;	&check;	&check;	&check;	&check;	&check;	&check;	&check;
U component of wind, u ($ms^{-1}$)	&check;	&check;	&check;	&check;	&check;	&check;	&check;	&check;	&check;	&check;
V component of wind, v ($ms^{-1}$)	&check;	&check;	&check;	&check;	&check;	&check;	&check;	&check;	&check;	&check;
Vertical velocity, w ($Pas^{-1}$)					&check;

1._LRA5 Reanalysis_ for Terrestrial (1979-2023)

Acronyms	Long Name	Units
asn	snow albedo	(0 - 1)
d2m	2-meter dewpoint temperature	K
e	total evaporation	m of water equivalent
es	snow evaporation	m of water equivalent
evabs	evaporation from bare soil	m of water equivalent
evaow	evaporation from open water	m of water equivalent
evatc	evaporation from top of canopy	m of water equivalent
evavt	evaporation from vegetation transpiration	m of water equivalent
fal	forecaste albedo	(0 - 1)
lai\_hv	leaf area index, high vegetation	$m^2 m^{-2}$
lai\_lv	leaf area index, low vegetation	$m^2 m^{-2}$
pev	potential evaporation	m
ro	runoff	m
rsn	snow density	$kg m^{-3}$
sd	snow depth	m of water equivalent
sde	snow depth water equivalent	m
sf	snowfall	m of water equivalent
skt	skin temperature	K
slhf	surface latent heat flux	$J m^{-2}$
smlt	snowmelt	m of water equivalent
snowc	snowcover	\%
sp	surface pressure	Pa
src	skin reservoir content	m of water equivalent
sro	surface runoff	m
sshf	surface sensible heat flux	$J m^{-2}$
ssr	net solar radiation	$J m^{-2}$
ssrd	download solar radiation	$J m^{-2}$
ssro	sub-surface runoff	m
stl1	soil temperature level 1	K
stl2	soil temperature level 2	K
stl3	soil temperature level 3	K
stl4	soil temperature level 4	K
str	net thermal radiation	$J m^{-2}$
strd	downward thermal radiation	$J m^{-2}$
swvl1	volumetric soil water layer 1	$m^3 m^{-3}$
swvl2	volumetric soil water layer 2	$m^3 m^{-3}$
swvl3	volumetric soil water layer 3	$m^3 m^{-3}$
swvl4	volumetric soil water layer 4	$m^3 m^{-3}$
t2m	2-meter temperature	K
tp	total precipitation	m
tsn	temperature of snow layer	K
u10	10-meter u-wind	$ms^{-1}$
v10	10-meter v-wind	$ms^{-1}$

1._ORAS Reanalysis_ for Sea-Ice (1979-2023)

Acronyms	Long Name	Units
iicethic	sea ice thickness	m
iicevelu	sea ice zonal velocity	$ms^{-1}$
iicevelv	sea ice meridional velocity	$ms^{-1}$
ileadfra	sea ice concentration	(0-1)
so14chgt	depth of 14$^\circ$ isotherm	m
so17chgt	depth of 17$^\circ$ isotherm	m
so20chgt	depth of 20$^\circ$ isotherm	m
so26chgt	depth of 26$^\circ$ isotherm	m
so28chgt	depth of 28$^\circ$ isotherm	m
sohefldo	net downward heat flux	$W m^{-2}$
sohtc300	heat content at upper 300m	$J m^{-2}$
sohtc700	heat content at upper 700m	$J m^{-2}$
sohtcbtm	heat content for total water column	$J m^{-2}$
sometauy	meridonial wind stress	$N m^{-2}$
somxl010	mixed layer depth 0.01	m
somxl030	mixed layer depth 0.03	m
sosaline	salinity	Practical Salinity Units (PSU)
sossheig	sea surface height	m
sosstsst	sea surface temperature	$^\circ C$
sowaflup	net upward water flux	$kg/m^2/s$
sozotaux	zonal wind stress	$N m^{-2}$

💡 Baseline Models

In addition to climatology and persistence, we evaluate the following:

1._Physics-based models (including control/perturbed forecasts)_:
2.[x] UKMO: UK Meteorological Office
3.[x] NCEP: National Centers for Environmental Prediction
4.[x] CMA: China Meteorological Administration
5.[x] ECMWF: European Centre for Medium-Range Weather Forecasts
6._Data-driven models_:
7.[x] Lagged-Autoencoder
8.[x] Fourier Neural Operator (FNO)
9.[x] ResNet
10.[x] UNet
11.[x] ViT/ClimaX
12.[x] PanguWeather
13.[x] GraphCast
14.[x] Fourcastnetv2

🏅 Evaluation Metrics

We divide our metrics into 3 classes: (1) Deterministic-based, which cover evaluation used in conventional deterministic forecasting tasks, (2) Physics-based, which are aimed to construct a more physically-faithful and explainable data-driven forecast, and (3) Probabilistic-based, which account for the skillfulness of ensemble forecasts.

1._Deterministic-based:_
2.[x] RMSE
3.[x] Bias
4.[x] Anomaly Correlation Coefficient (ACC)
5.[x] Multiscale Structural Similarity Index (MS-SSIM)
6._Physics-based:_
7.[x] Spectral Divergence (SpecDiv)
8.[x] Spectral Residual (SpecRes)

1._Probabilistic-based:_
2.[x] RMSE Ensemble
3.[x] Bias Ensemble
4.[x] ACC Ensemble
5.[x] MS-SSIM Ensemble
6.[x] SpecDiv Ensemble
7.[x] SpecRes Ensemble
8.[x] Continuous Ranked Probability Score (CRPS)
9.[x] Continuous Ranked Probability Skill Score (CRPSS)
10.[x] Spread
11.[x] Spread/Skill Ratio