Modelpublic

OneScience-Group/CorrDiff

sourceHugging Faceapache-2.0updated 21d agoView on Hugging Face

0likes32downloads

Model Card

CorrDiff

Model Introduction

CorrDiff is a two-stage generative model for kilometer-scale atmospheric downscaling. It first predicts a high-resolution mean with conditional regression and then uses a residual diffusion model to generate local stochastic details and ensemble forecasts.

Paper: Residual Corrective Diffusion Modeling for Km-scale Atmospheric Downscaling https://arxiv.org/abs/2309.15214

Model Description

CorrDiff was proposed by an NVIDIA research team. The model is trained using ERA5 reanalysis data and high-resolution regional CWA-WRF simulation data, and performs weather-field downscaling through regression and residual diffusion. It is suitable for high-resolution weather-field generation, kilometer-scale downscaling, and ensemble uncertainty analysis.

Use Cases

Scenario	Description
Weather downscaling	Downscale `36x36` coarse-resolution condition fields to `448x448`.
Ensemble forecasting	Generate multiple possible high-resolution results through diffusion sampling.
Radar reflectivity generation	Generate high-resolution maximum radar reflectivity that is absent from the ERA5 condition fields, supporting analysis of fine-scale precipitation-system structures.
Extreme-weather risk analysis	Use ensemble members and spatial uncertainty to characterize the possible evolution of local heavy precipitation and other high-impact weather.
Local engineering validation	Use a small amount of synthetic data to check training, inference, and evaluation workflows.
Multi-GPU training	Launch distributed training with `torchrun`.

Usage Guide

1.OneCode

Experience intelligent one-click AI4S programming through the OneCode online environment:

Click to Experience Intelligent One-Click AI4S Programming

2.Download and Installation

bash

hf download OneScience-Group/CorrDiff --local-dir ./CorrDiff
cd CorrDiff

Environment Dependencies

Hardware Requirements

—A GPU or DCU is recommended.
—CPU can be used for connectivity validation with a small configuration; full training and inference are slower.
—DCU users must install DTK in advance. DTK 25.04.2 or above, or the OneScience-recommended version matching the current cluster, is recommended.

DCU Environment

bash

# Please activate DTK and CONDA first
conda create -n onescience311 python=3.11 -y
conda activate onescience311
# uv installation is supported
pip install onescience[earth-dcu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai

GPU Environment

bash

# Please activate CONDA first
conda create -n onescience311 python=3.11 -y libstdcxx-ng=12 libgcc-ng=12 gcc_linux-64=12 gxx_linux-64=12
conda activate onescience311
# uv installation is supported
pip install onescience[earth-gpu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai

Training Data Introduction

By default, two synthetic samples following the same protocol are used to validate the engineering workflow. The data are saved as data/corrdiff.npz, and the synthetic data do not represent real weather distributions.

The synthetic data retain the input and output specifications of the Taiwan downscaling experiment in the paper: 12-channel 36x36 condition fields and 4-channel 448x448 target fields.

Real data must be preprocessed and converted to the following NPZ training protocol. This protocol is consistent with the model input specification but is not equivalent to the original datasets' download format.

text

input: float32 [N,12,36,36]
target: float32 [N,4,448,448]

fake_data.py automatically writes the protocol and data_source protocol metadata. These fields must be retained when using real data.

bash

python scripts/fake_data.py

Training

bash

python scripts/train.py

Multi-GPU training can use:

bash

torchrun --nproc_per_node=8 scripts/train.py

Training sequentially optimizes the conditional-regression and residual-diffusion stages and saves a checkpoint and overall training metrics. The default configuration is intended for quick workflow validation. Formal experiments should use the data scale, model configuration, and training schedule corresponding to the paper.

text

result/checkpoints/corrdiff.pt
result/training/metrics.json

Training Weights

This repository will provide CorrDiff training weights in the weight/ folder. The weight files will be uploaded soon and are expected to be available in the near future.

Inference

bash

python scripts/inference.py

Inference loads the training checkpoint, generates high-resolution ensemble forecasts according to the configuration, and saves the results to:

text

result/output/predictions.npz

Evaluation and Visualization

bash

python scripts/result.py

Evaluation reports overall MAE, RMSE, ensemble CRPS, and ensemble spread, and generates an ensemble diagnostic figure. Synthetic-data results are only for validating the engineering workflow and do not represent the full performance reported in the paper.

text

result/evaluation/metrics.json
result/evaluation/ensemble_diagnostics.png

Official OneScience Resources

Platform	OneScience Main Repository	Skills Repository
Gitee	https://gitee.com/onescience-ai/onescience	https://gitee.com/onescience-ai/oneskills
GitHub	https://github.com/onescience-ai/OneScience	https://github.com/onescience-ai/oneskills

Citation and License

This repository is a reproduction of the original CorrDiff paper.

The use of the code and data in this repository remains subject to the licenses and terms of use of their respective projects.