jcamier/met-office-uk-deterministic-zarr

Met Office UK Deterministic Dataset (Zarr Format)

Description

This dataset is a subset of the Met Office UK Deterministic Dataset, converted from the original NetCDF format into Zarr format for modern data analysis. The Zarr files are packaged as tar archives for efficient storage and transfer.

The subset focuses on specific variables and configurations, which are detailed in the met_office_uk_data_config.yaml file included in this repository. Researchers and developers can use this subset for applications in climate science, weather forecasting, and renewable energy modeling.

Usage

This dataset is provided under the Creative Commons Attribution 4.0 International License (CC-BY-4.0). When using this dataset, you must provide proper attribution to the Met Office as outlined below.

Attribution

This dataset is derived from the Met Office UK Deterministic Dataset, which is British Crown copyright and provided by the Met Office under the terms of the UK Open Government License (OGL).

• —Original Source: Met Office UK Deterministic Dataset on AWS
• —Copyright: British Crown copyright © Met Office

Details

Zarr Format and Tar Archives

• —Format: The dataset files are in Zarr format, a modern storage format optimized for analytics and machine learning workflows.
• —Packaging: Zarr files are stored as .tar.gz archives. Each archive corresponds to a specific time interval, such as 2022-12-01-00.tar.gz.

Subset Configuration

• —The dataset includes a subset of variables and configurations chosen for targeted use cases.
• —For details on the variables and configurations included in this dataset, please refer to the `met_office_uk_data_config.yaml` file.

How to Access

You can download and extract the tar archives using the following Python snippet:

import tarfile

# Example: Extracting a tar.gz archive
archive_path = "2022-12-01-00.tar.gz"
with tarfile.open(archive_path, "r:gz") as tar:
    tar.extractall(path="extracted_zarr")