risenyard/egms-qa-dataset

EGMS-QA Dataset

Prepared EGMS displacement tiles, encoder tokens, task labels, reference tables, and natural-language QA records for 10,000 overlapping 7 km tiles. This card describes the available data, file formats, and download options.

<a id="use-the-data"></a>

Data access

The links above open the published files and directories. For code installation and use, follow the GitHub module guides.

QA use

Install a current version of datasets:

pip install -U datasets

Read a QA record without downloading the full release:

from datasets import load_dataset

qa = load_dataset("risenyard/egms-qa-dataset", streaming=True)
print(next(iter(qa["test"])))

Streaming reads QA records on demand. Omit streaming=True to download the three QA splits. Tiles, tokens, and task tables are separate files.

<a id="release-layers"></a> <a id="data-requirements-by-component"></a>

Data files

Encoder data

Available files include NPZ tiles, the split manifest, and data configuration. See the source-tile format and normalization.

Translator data

Use the token cache, labels and task metadata, reference tables, and approved phrasings. Use tile_id to match features and targets; retain each tile's split assignment. See token fields and shapes.

QA construction data

Use labels, reference tables, and QA metadata. Recomputing selected tasks may also require source tiles or tokens. Input requirements and construction commands are listed in the GitHub QA guide.

Data contracts

Shared identifiers and splits

Use tile_id to join tiles, tokens, labels, and QA. The split manifest records tile IDs, source paths, and split assignments. Token-cache rows are identified by tile_ids, with matching entries in splits.

Splits are fixed at tile level. Tiles overlap, so a persistent-scatterer ID may occur in more than one tile.

Source-tile contract

The NPZ files contain processed measurements from the EGMS Level-3 Ortho Vertical product for 2019–2023.

To compute task values for new tiles, include the eight EGMS point attributes height, rmse, mean_velocity, mean_velocity_std, acceleration, acceleration_std, seasonality, and seasonality_std, each as a point-aligned [N] array. Stored indices [0,294) correspond exactly to [8,302) on the 304-step source-preparation axis. The data config preserves the source offset and six-day cadence.

Encoder representations

The cache contains 65 tokens of width 256 per tile: one summary token followed by 64 spatial-cell tokens in row-major order.

The cache metadata records the layout, source-data contract, and input checksums.

QA records

Each record pairs a tile and task with a question phrasing and reference answer. Task definitions are listed in the GitHub task catalog.

The files in data/qa/ are the fixed published splits. Newly generated QA may use different phrasings, sampling settings, or record order.

Labels and task metadata

labels.parquet contains one row per tile: tile_id, split, and 64 task-value columns such as B21 and S31. Numeric values use the task's documented units; categorical values use its defined labels. Null values indicate unavailable targets.

metadata.json maps task IDs to source tables and columns and records target types and coverage. The 14 X refusal tasks are stored as task-level catalogs rather than tile-dependent label columns.

Reference tables

Each of the 27 task groups has a <group>_final_table.csv under `artifacts/reference_tables/`. A/B/C/D/S tables contain tile-level values keyed by tile_id and split; X1–X3 contain refusal catalogs. Definitions and computational methods are listed in the GitHub task index.

Normalization contract

normalization.json records the displacement mean, standard deviation, and linear-detrended residual standard deviation, fitted on the 8,000 training tiles over stored indices [0,294).

Metadata

The checksum inventory excludes .gitattributes, metadata/files.sha256, and metadata/release_manifest.json. The latter two describe the inventory and are excluded to avoid self-referential hashes.

Provenance, terms, and limitations

Generated using European Union's Copernicus Land Monitoring Service information; https://doi.org/10.2909/4a14a29b-7db7-40e4-81ad-df3aa8dfbc6f

The EGMS measurements were selected, arranged into overlapping 7 km research tiles, and combined with EGMS-QA-created labels and questions. This is not an official EU, EEA, Copernicus, or EGMS product; no endorsement is implied.

The QA records, derived labels/tables, and token cache are CC-BY-4.0. The Copernicus-derived measurements retain the CLMS attribution and modification conditions. See DATA_TERMS.md and SOURCE_PROVENANCE.md.

The data describe measured deformation histories. They do not establish cause, forecast future motion, certify structural safety, or replace expert InSAR and geotechnical assessment.