LordZeee/ndvi-convlstm

ConvLSTM model for NDVI forecasting in Ukraine

1. Data Generation & Sampling Strategy

This section details the "what, where, when, and how" of the data extraction process from Google Earth Engine.

Geographic & Temporal Focus

• —Regions of Interest: The data is sampled from three 1°x1° bounding boxes located in major agricultural oblasts of Ukraine:
• —Cherkasy_Croplands_Initial (Central Ukraine)
• —Vinnytsia_Croplands_Initial (Central-Western Ukraine)
• —Mykolaiv_Croplands_Initial (Southern Ukraine)
• —Time Period: Data is collected from 2018 to 2021. This timeframe was specifically chosen to establish a pre-war baseline of agricultural activity.
• —Growing Season: Sampling is limited to the typical growing season, from April (`season_start_month: 4`) to October (`season_end_month: 10`).
• —Prediction Task: The model is trained to predict 4 future weeks of data (pred_len: 4) using the previous 6 weeks as input (input_len: 6).

Spatial Sampling (Minicubes)

Data is not downloaded as continuous maps but as small spatio-temporal samples called "minicubes."

• —Spatial Resolution: 20 meters per pixel, based on Sentinel-2's native resolution.
• —Minicube Size: 64x64 pixels, which corresponds to a physical area of 1.28 km x 1.28 km.
• —Sampling Density: The script aims to generate 100 high-quality minicube time-series per ecoregion.

Data Quality Control

To ensure only high-quality, cloud-free data is used for training, a two-step filtering process is applied:

1. 1.Initial Location Selection: A minicube location is only considered valid if:
2. 2.At least 70% of its area is land (min_land_pixel_percentage).
3. 3.At least 50% of its area has valid NDVI data during a quality check period (min_valid_ndvi_pixel_percentage).
4. 4.The land cover consists of valid classes, including croplands, trees, shrublands, and sparse vegetation (valid_esa_wc_classes: [10, 20, 30, 40, 60, 90]).
5. 5.Time-Series Sequence Filtering: After locations are chosen, their entire time-series is evaluated. A full 10-week sequence (6 input + 4 prediction) is only exported if:
6. 6.It contains at least 10 "good" weeks (min_good_weeks_per_sequence).
7. 7.A "good" week is defined as having, on average, at least 80% clear pixels contributing to the weekly NDVI composite (min_avg_clear_pixels_for_good_week).

2. Data Sources & GEE Assets

The dataset is built by integrating information from multiple satellite and climate data products available in Google Earth Engine.

• —Primary Optical Imagery: COPERNICUS/S2_SR_HARMONIZED (Sentinel-2)
• —Weather Data: ECMWF/ERA5_LAND/HOURLY
• —Land Cover: ESA/WorldCover/v200 & ESA/WorldCover/v100
• —Topography: USGS/SRTMGL1_003 (Digital Elevation Model)
• —Ancillary Vegetation: MODIS/061/MOD15A2H (LAI/FAPAR)
• —Soil Properties: OpenLandMap SoilGrids (Clay, Sand, Organic Carbon)
• —Surface Water: JRC/GSW1_4/GlobalSurfaceWater

3. Model Input Features

The model is trained on a rich set of 20 features, combining dynamic (time-varying) and static (constant) data.

• —Target Variable: The model's objective is to predict `NDVI`.
• —Input Features: The final features used for training (final_input_band_names_for_npz) are listed below.

4. Model & Training Configuration

Model Architecture (ConvLSTM)

The forecasting model is a Convolutional LSTM (ConvLSTM) network, well-suited for spatio-temporal data.

• —Layers: 3 ConvLSTM layers.
• —Hidden Dimensions: The number of channels in each layer is [64, 128, 64].
• —Kernel Sizes: A 3x3 convolutional kernel is used in all layers.
• —Attention Mechanism: A self-attention mechanism is enabled (use_attention: True) to help the model focus on more relevant spatial features.

Training Parameters

• —Device: cuda (GPU)
• —Epochs: 30
• —Batch Size: 8
• —Learning Rate: Initial 0.001, with a ReduceLROnPlateau scheduler that reduces the rate if validation loss stagnates for 7 epochs.
• —Validation Split: 20% of the data is held out for validation.
• —Sample Weighting: Enabled (use_sample_weighting: True). Training samples are weighted based on the average clarity (NDVI_clear_pixel_count) of their input weeks, giving more importance to less cloudy samples.

5. File Directories

This configuration specifies paths for data handling, from initial GEE export to final model artifacts.

• —GEE Export Folder (Drive): DT_NDVI_Ukraine_01
• —Downloaded Raw TFRecords: ./data/downloaded_tfrecords_03
• —Processed `.npz` Training Data: ./data/processed_training_data_03
• —Saved Models & Outputs: ./model_outputs/config_003/
• —Prediction Visualizations: ./img/config_003/