ai-spatial/DERE

DERE Dataset

DERE is a multi-source ecosystem dataset for global carbon-flux prediction. It integrates Ecosystem Demography (ED) simulations, ED-derived vegetation structure, ESA CCI plant functional type fractions, LiDAR-derived forest-age information, and real-world in-situ carbon-flux observations.

<p align="center"> <img src="./DEREDatasetOverview.png" alt="DERE Dataset Overview" width="1000"> </p>

The dataset is organized into two complementary collections. GlobalMask provides globally sampled simulation and remote-sensing data, while InSituMatched links the same simulation and auxiliary information with observed GPP, RECO, and NEE from multiple flux-tower networks. Together, they support knowledge-guided learning, multi-source data fusion, process-model emulation, and simulation-to-observation evaluation.

The dataset supports our KDD 2026 paper:

Knowledge-Guided Learning for Global Carbon Flux Prediction: Integrating High-Level Remote Sensing with Bottom-Up Physical Modeling

Code repository: https://github.com/ai-spatial/DERE

Dataset Viewer

The Hugging Face Dataset Viewer uses lightweight Parquet summary tables under viewer/. Each Viewer row corresponds to one sample in the associated NPZ file and includes sample identifiers, array shapes, file paths, and compact summary statistics.

The complete multidimensional arrays remain in the NPZ files under GlobalMask/ and InSituMatched/.

Dataset organization

DERE/
├── README.md
├── CITATION.cff
├── GlobalMask/
│   ├── README.md
│   ├── train/
│   │   └── GlobalMask_train.npz
│   └── test/
│       └── GlobalMask_test.npz
├── InSituMatched/
│   ├── README.md
│   ├── above/
│   │   ├── train/
│   │   │   └── InSituMatched_above_train.npz
│   │   └── test/
│   │       └── InSituMatched_above_test.npz
│   ├── ameriflux/
│   │   ├── train/
│   │   │   └── InSituMatched_ameriflux_train.npz
│   │   └── test/
│   │       └── InSituMatched_ameriflux_test.npz
│   ├── fluxnet/
│   │   ├── train/
│   │   │   └── InSituMatched_fluxnet_train.npz
│   │   └── test/
│   │       └── InSituMatched_fluxnet_test.npz
│   ├── icos_ww/
│   │   ├── train/
│   │   │   └── InSituMatched_icos-ww_train.npz
│   │   └── test/
│   │       └── InSituMatched_icos-ww_test.npz
│   └── multiple/
│       ├── train/
│       │   └── InSituMatched_multiple_train.npz
│       └── test/
│           └── InSituMatched_multiple_test.npz
├── metadata/
│   ├── dimension_definitions.md
│   ├── normalization_statistics.npz
│   ├── feature_names.csv
│   ├── target_names.csv
│   ├── pft_names.csv
│   ├── age_classes.csv
│   ├── insitu_site_metadata.csv
│   ├── source_licenses.csv
│   ├── train_test_mask.md
│   └── train_test_mask.npy
├── scripts/
│   └── create_viewer_tables.py
└── viewer/
    ├── README.md
    ├── global_mask/
    │   ├── train.parquet
    │   └── test.parquet
    └── insitu_matched/
        ├── above/
        ├── ameriflux/
        ├── fluxnet/
        ├── icos-ww/
        └── multiple/

GlobalMask

GlobalMask contains globally sampled land-grid cells selected by a fixed train/test mask.

Each file contains:

• —ed_simulation_x
• —ed_simulation_y
• —ed_simulation_pft_bl
• —ed_simulation_pft_nl
• —ed_simulation_pft_gs
• —lidar_age_weight_fraction
• —esa_cci_bl_fraction
• —esa_cci_nl_fraction
• —esa_cci_gs_fraction

InSituMatched

InSituMatched contains ED simulation data and auxiliary variables aligned with in-situ carbon-flux observations.

Each file contains:

• —ed_simulation_x
• —ed_simulation_y
• —observed_y
• —lidar_age_weight_fraction
• —esa_cci_bl_fraction
• —esa_cci_nl_fraction
• —esa_cci_gs_fraction

The multiple subset contains sites represented in more than one in-situ network. These sites are separated from the network-specific subsets to avoid overlap between subsets.

Temporal alignment

The complete ED target sequence covers 29 calendar years from 1992 through 2020.

• —December 1992 is used as the initial ED state.
• —Model inputs cover January 1993 through December 2020.
• —Prediction targets cover January 1993 through December 2020.
• —The prediction period contains 28 years, or 336 monthly time steps.

All released arrays use sample-first orientation whenever a sample dimension is present. Detailed dimensions and released shapes are documented in metadata/dimension_definitions.md.

Metadata

• —feature_names.csv defines the ordering and index ranges of the 136 ED input features.
• —target_names.csv defines the 10 ED simulation targets and 3 observed carbon-flux targets.
• —pft_names.csv defines broadleaf, needleleaf, and grass-and-shrub PFTs.
• —age_classes.csv defines the 18 representative forest-age classes.
• —insitu_site_metadata.csv maps each InSituMatched sample index to its network and split.
• —source_licenses.csv documents the sources, licenses, redistribution terms, and citation requirements of the released data components.
• —train_test_mask.md documents the GlobalMask spatial mask.
• —train_test_mask.npy stores the GlobalMask sampling split.
• —normalization_statistics.npz contains:
• —x_mean: shape [136]
• —x_std: shape [136]
• —y_mean: shape [10]
• —y_std: shape [10]

Loading the data

import numpy as np

file_path = "GlobalMask/train/GlobalMask_train.npz"

with np.load(file_path, allow_pickle=False) as data:
    for key in data.files:
        print(key, data[key].shape, data[key].dtype)

Load the normalization statistics with:

import numpy as np

with np.load(
    "metadata/normalization_statistics.npz",
    allow_pickle=False,
) as stats:
    x_mean = stats["x_mean"]
    x_std = stats["x_std"]
    y_mean = stats["y_mean"]
    y_std = stats["y_std"]

Standardization is performed as:

x_normalized = (x - x_mean) / x_std
y_normalized = (y - y_mean) / y_std

Intended use

The dataset is intended for research on:

• —global carbon-flux prediction
• —knowledge-guided machine learning
• —process-model emulation
• —multi-source data fusion
• —time-series modeling of GPP, RECO, and NEE
• —simulation-to-observation transfer learning
• —reproduction and comparison of DERE and baseline models

Citation

If you use this dataset, please cite:

@inproceedings{xu2026knowledge,
  author    = {Shuo Xu and Zhihao Wang and Ruohan Li and Ruichen Wang and Lei Ma and George C. Hurtt and Xiaowei Jia and Yiqun Xie},
  title     = {Knowledge-Guided Learning for Global Carbon Flux Prediction: Integrating High-Level Remote Sensing with Bottom-Up Physical Modeling},
  booktitle = {Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2},
  year      = {2026},
  address   = {Jeju Island, Republic of Korea},
  publisher = {ACM},
  doi       = {10.1145/3770855.3818927}
}

The same citation is also provided in CITATION.cff.

License and source terms

The released files combine information derived from multiple upstream sources. Users are responsible for following the applicable attribution and redistribution terms of those sources.