cangyeone/SeismicX-Cont

<p align="center"> <img src="logo.png" alt="SeismicX logo" width="180"/> </p>

<h1 align="center">SeismicX-Cont — HDF5 Continuous-Waveform Data Product</h1>

<p align="center"> Processed continuous-waveform database for reproducible regional monitoring studies </p>

1. Overview

SeismicX-Cont is a processed and structured HDF5-based continuous-waveform data product for deployment-level earthquake-monitoring studies.

The release preserves continuous waveform context, station-time coverage, annotation provenance, instrument-response metadata, and validation lineage in a common file organization. It is intended as a reusable monitoring data product, not as a relative model-assessment resource or a replacement for the authoritative SCEDC, NCEDC, or CEED source archives.

The repository includes the SeismicX DataLoader, SQLite coverage-query utilities, validation scripts, and quick-start notebooks for reproducible access to the released files.

Dataset preview

The Hugging Face Dataset Viewer displays a lightweight file index rather than the full HDF5 waveform contents. The waveform data are stored as structured HDF5 files and should be accessed using the provided documentation and scripts. The preview table is intended only to summarize available files, time periods, networks, and data types.

1.1 Repository Layout

The repository is organized as follows:

data/
  hdf5/                 # 14 daily HDF5 waveform files
  index/                # SQLite waveform index
  label/                # annotations, reference tables, optional consensus and multi-output catalogue
  response/             # instrument-response JSON for dataloader correction/simulation
  validation/           # machine-readable audits and optional phase-matching diagnostics
notebooks/              # interactive quickstart notebook
pickers/                # example TorchScript/ONNX picker models
scripts/                # conversion, indexing, picker, association, and upload scripts
essd_scripts/           # ESSD statistics, diagnostics, and plotting scripts
utils/                  # reusable HDF5 dataloader and waveform-index APIs

The main data-preparation and picker entry points live in scripts/; ESSD manuscript numbers, statistics, diagnostics, and plotting entry points live in essd_scripts/. Use essd_scripts/audit_manuscript_numbers.py to regenerate inventory and exact NSLC point-coverage values, including finite-sample checks for floating-point HDF5 arrays, independently of all picker outputs. Use essd_scripts/build_reference_arrivals.py to regenerate C0 point coverage and configuration-specific C1--C3 window, component, gap, sampling-rate, and response fields in data/label/reference_arrivals.sqlite. The database preserves all source readings in reference_arrivals, stores one canonical station-location-phase-time denominator row in evaluation_references, and links every canonical row back to its source arrivals and event identifiers through evaluation_reference_members. The released configuration selects one location and channel family deterministically, never combines components across locations or families, and uses half-open response epochs. Response-qualified C3 requires exactly one matching response epoch for every selected component. Use essd_scripts/audit_waveform_quality.py for the full index-integrity audit and the deterministic period/network/channel-family-stratified HDF5 sample reported in the manuscript. Canonical machine-readable summaries are released as data/validation/manuscript_number_audit.json and data/validation/waveform_quality_audit.json; the reproduction script refreshes them from the distributed data files. The optional data/validation/multimodel_phase_matching/ extension stores automatically regenerated one-to-one phase-agreement records and summaries for non-standardized example outputs. Regenerate it with ./essd_scripts/regenerate_multimodel_phase_matching.sh; it is an automated consistency and interoperability check, not a model comparison or ground truth. The script reads versioned C0 flags from reference_arrivals.sqlite, which was independently generated from exact NSLC intervals and finite HDF5 samples. The optional compressed multi-output catalogue under data/label/ retains every source pick from nine non-standardized example runs and groups station-phase records with a strict span below 1.5 s. It provides compact JSONL and normalized SQLite representations for querying output agreement. The adjacent output_eligibility.sqlite records processed station-time intervals, and output_registry.tsv documents the nine source output streams. The released support-stratified validation tables report candidate-centred agreement and manual-reference recovery, both before and after normalization by the number of eligible outputs. These summaries support the two explicitly conditioned phase-pick precision proxies described in the manuscript; they do not claim an exhaustive conventional full-stream precision measurement. Output support is not ground truth or a relative model assessment. Workflow-level results are released in data/validation/leave_one_output_precision_proxy.tsv, including candidate reconstruction after removing each assessed output and a 20-replicate station-day time-shift null for the declared example output. The catalogue-conditioned window sensitivity is released in data/validation/reference_window_precision_proxy_sensitivity.tsv for T=1,5,10,30 s. A collection of serialized picker artifacts used by the project is provided under pickers/. The PNSN implementation and related phase-detection models are additionally archived at https://doi.org/10.5281/zenodo.17510174, and the PNSN workflow is described at https://doi.org/10.1029/2025JH000944. See pickers/README.md for the output-to-artifact mapping and data/label/README_all_model_phase_picks.md for the candidate schema and query examples. The quickstart notebook notebooks/seismicx_cont_quickstart.ipynb demonstrates file checks, SQLite queries, HDF5 waveform plotting, annotation inspection, and evaluation command templates.

Current scripts/ contents:

Release metadata and integrity files:

• —VERSION records the release version (v1.0.0).
• —CITATION.cff provides machine-readable citation metadata.
• —LICENSE_CLARIFICATION.md explains the layer-specific data-use terms.
• —manifest_sha256.txt lists release-file SHA-256 checksums.
• —manifest_filesizes.tsv lists release-file sizes in bytes.
• —Full release: https://huggingface.co/datasets/cangyeone/SeismicX-Cont. The version-specific DOI for the frozen submission release is cited in the ESSD manuscript and on the Hugging Face DOI panel.
• —Mini two-hour dataset DOI (Zenodo): https://doi.org/10.5281/zenodo.21472154
• —Mini two-hour dataset DOI (Hugging Face): https://doi.org/10.57967/hf/9664 (Revision 883e092)
• —Public mirror: https://www.modelscope.cn/datasets/cangyeone/SeismicX-Cont

Recommended quickstart and reproducibility order:

1. 1.Inspect the repository layout and file checks with notebooks/seismicx_cont_quickstart.ipynb.
2. 2.Verify the full release structure with python scripts/verify_full_dataset.py.
3. 3.Build or refresh the SQLite waveform index with ./scripts/build_waveform_index.sh when rebuilding from HDF5 files.
4. 4.Regenerate the release checksum and file-size manifests with ./scripts/write_manifest.sh.
5. 5.Reproduce ESSD manuscript tables and figures with ./essd_scripts/reproduce_manuscript_outputs.sh.
6. 6.If the optional source picker JSONL files are present, regenerate the multi-output agreement layer with ./essd_scripts/regenerate_multimodel_phase_matching.sh.
7. 7.Query the optional multi-output candidate catalogue after decompressing data/label/all_model_phase_picks.sqlite.zst; schema and examples are in data/label/README_all_model_phase_picks.md.
8. 8.Use data/label/output_eligibility.sqlite for processed-domain queries and inspect data/validation/multi_output_manual_reference_match.tsv and data/validation/multi_output_eligibility_normalized_match.tsv for the released support-stratified catalogue-agreement summaries.
9. 9.Inspect data/validation/leave_one_output_precision_proxy.tsv and data/validation/reference_window_precision_proxy_sensitivity.tsv for the workflow-level and reference-window precision-proxy results.

A tested environment is macOS 26.5.2 on Apple silicon with Python 3.12.2. Exact versions of the principal runtime packages are listed in requirements.txt. Install them in an isolated environment with python -m pip install -r requirements.txt. The released scripts use only the Python standard library, the listed third-party packages, and modules distributed under scripts/, essd_scripts/, and utils/; no unpublished local Python module is required for the documented access and validation paths.

2. Dataset Description

2.1 Data Source

This dataset contains continuous waveform records from three regional seismic networks:

• —CI — Southern California Seismic Network
• —NC — Northern California Seismic Network
• —BK — Berkeley Digital Seismic Network

These networks present varied observation conditions, including differences in instrument response, channel configuration, noise characteristics, and station density.

2.2 Time Windows

Two representative 7-day continuous time windows are provided:

TIME_WINDOWS = [
    {
        "id": "day_20190701",
        "starttime": "2019-07-01T00:00:00",
        "endtime": "2019-07-08T00:00:00",
    },
    {
        "id": "day_20211108",
        "starttime": "2021-11-08T00:00:00",
        "endtime": "2021-11-15T00:00:00",
    },
]

Each window covers 7 consecutive days of continuous seismic recordings.

2.3 Data Content

The dataset includes continuous three-component waveform data in HDF5 format, station metadata (latitude, longitude, elevation, validity period), multi-segment waveform records merged into continuous traces, and event annotations derived from the CEED dataset.

3. Monitoring-Period Design

SeismicX-Cont pairs two one-week monitoring periods in the same HDF5, annotation, response, and coverage-index structure. The paired design supports reusable continuous-stream validation under contrasting monitoring conditions.

3.1 High-Activity Window (July 2019)

Dense seismicity with frequent and overlapping events. Reuse typically focuses on coverage-qualified recall, phase-time residuals, station coverage, and the ability to inspect workflow behavior during clustered earthquake sequences.

3.2 Low-Activity Window (November 2021)

Sparse seismicity with extended quiet periods relative to the Ridgecrest week. Reuse typically focuses on catalog-unmatched automatic-pick burden, noise conditions, station availability, and reporting stability over long continuous intervals. Catalog-unmatched picks should not be interpreted automatically as confirmed false detections.

4. Data Structure

The dataset is stored in a hierarchical HDF5 layout:

/year
  /day
    /stations
      /network.station.location
        /waveform
          /channel
            /segment

Example:

/2019-01-01T00:00:00.000000Z
  /2019-07-01T00:00:00.000000Z
    /stations
      /BK.BDM.00
        /waveform
          /BHE/0
          /BHN/0
          /BHZ/0

Each channel can contain multiple waveform segments that are automatically merged during loading.

5. SeismicX DataLoader

File: utils/hdf5_waveform_dataset.py

A PyTorch Dataset that wraps the hierarchical HDF5 files and delivers ready-to-use waveform tensors with full station metadata. Designed for single-pass inference over large multi-file datasets with minimal memory footprint.

5.1 Key Features

Multi-file input — accepts a single file, a directory, a glob pattern (data/hdf5/*.h5), or a Python list of paths.

Channel-family grouping — waveforms are grouped by the first two characters of the channel code (the "family"), so BHE/BHN/BHZ form one sample rather than three separate ones.

Automatic 3-component construction — E/1 → slot 0, N/2 → slot 1, Z/3 → slot 2. Z-only stations can be replicated to [Z, Z, Z].

Multi-segment merging — consecutive HDF5 segments are merged into one contiguous trace with gap-filling.

Optional resampling — linear-interpolation resampling to a target sampling rate (no scipy dependency).

Optional instrument-response handling — the loader can remove the native instrument response using data/response/instrument_responses.json and can optionally simulate a user-selected target response.

Memory-safe file handle caching — only the current HDF5 file's handle is kept open; when the loader moves to a new file the old handle (and its chunk cache) is closed immediately. Memory usage stays O(1) regardless of the number of files.

5.2 Constructor Parameters

HDF5WaveformDataset(
    h5_file,
    mode="three",
    fill_value=0.0,
    dtype=np.float32,
    default_location="--",
    allowed_families=("HH", "BH", "EH", "HN"),
    allowed_z_only_channels=("EHZ",),
    allow_z_only=True,
    replicate_z_only=False,
    target_sampling_rate=None,
    skip_sample_keys=None,
    skip_jsonl=None,
    skip_record_type="phase_pick",
    keep_h5_open=True,
    include_segments_metadata=True,
    use_overlap_mask=True,
    h5_rdcc_nbytes=8 * 1024 * 1024,
    max_duration_sec=90000.0,
    instrument_response_json=None,
    remove_instrument_response=False,
    response_output="VEL",
    response_pre_filt=None,
    response_water_level=60,
    response_error_behavior="raise",
    simulate_instrument_response=False,
    simulation_response_json=None,
    simulation_response_id=None,
    simulation_response_selector=None,
    simulation_paz=None,
    simulation_output=None,
)

5.3 Output Modes

mode="single"

One sample per channel. Output waveform shape: [T].

mode="three" (recommended)

One sample per channel family. Observed components are arranged as E/N/Z (or 1/2/3 mapped to the same slots). A Z-only record has shape [T, 1] by default; synthetic [Z,Z,Z] input requires replicate_z_only=True.

mode="multi"

One sample per channel family, all available channels stacked. Output waveform shape: [T, C].

Every mode returns validity_mask, which is false for gap-filled samples, and component_mask, which identifies observed output components. Three-component mode also returns observed_component_mask; replicated Z traces never become observed horizontal components in this mask. Resampling uses an anti-aliased polyphase filter when SciPy is available, and the validity mask is resampled conservatively so interpolation does not hide gaps.

5.4 Usage Examples

Single file:

from utils.hdf5_waveform_dataset import HDF5WaveformDataset, waveform_collate_fn

dataset = HDF5WaveformDataset(
    h5_file="data/hdf5/continuous_waveform_usa_20190701.h5",
    mode="three",
    target_sampling_rate=100.0,
)

Glob pattern (multiple files):

dataset = HDF5WaveformDataset(
    h5_file="data/hdf5/continuous_waveform_usa_*.h5",
    mode="three",
    allowed_families=("HH", "BH", "EH", "HN"),
    allow_z_only=True,
    replicate_z_only=True,
    target_sampling_rate=100.0,
)
print(f"Files: {len(dataset.h5_files)}  |  Samples: {len(dataset)}")

Directory:

dataset = HDF5WaveformDataset(h5_file="data/hdf5/", mode="three")

Remove native instrument response:

dataset = HDF5WaveformDataset(
    h5_file="data/hdf5/continuous_waveform_usa_20190701.h5",
    mode="three",
    instrument_response_json="data/response/instrument_responses.json",
    remove_instrument_response=True,
    response_output="VEL",
    response_pre_filt=(0.2, 0.5, 20.0, 45.0),
    response_water_level=None,
)

Remove the native response and simulate a target response:

dataset = HDF5WaveformDataset(
    h5_file="data/hdf5/continuous_waveform_usa_20190701.h5",
    mode="single",
    instrument_response_json="data/response/instrument_responses.json",
    remove_instrument_response=True,
    response_output="VEL",
    response_pre_filt=(0.2, 0.5, 20.0, 45.0),
    response_water_level=None,
    simulate_instrument_response=True,
    simulation_response_selector={
        "network": "BK",
        "station": "BDM",
        "location": "00",
        "channel": "BHZ",
        "time": "2019-07-06T04:00:00Z",
    },
    simulation_output="VEL",
)

The returned item includes an instrument_processing field with the matched native response_id, response epoch, simulated response id, and any processing error. Response removal and response simulation use ObsPy's response machinery; the dataloader only provides the SeismicX-Cont JSON lookup and per-sample matching layer.

5.5 DataLoader Configuration

Single-process (safest — required for MPS):

from torch.utils.data import DataLoader

loader = DataLoader(
    dataset,
    batch_size=1,
    shuffle=False,
    num_workers=0,
    collate_fn=waveform_collate_fn,
)

Multi-process (recommended for CUDA):

h5py is not fork-safe. On Linux the default DataLoader start method is fork, which can corrupt HDF5 reads. Use spawn explicitly:

from utils.hdf5_waveform_dataset import hdf5_worker_init_fn

loader = DataLoader(
    dataset,
    batch_size=4,
    shuffle=False,
    num_workers=4,
    collate_fn=waveform_collate_fn,
    multiprocessing_context="spawn",   # avoids h5py + fork deadlocks on Linux
    worker_init_fn=hdf5_worker_init_fn,
    prefetch_factor=2,
    persistent_workers=True,
    pin_memory=True,                   # speeds up CPU → GPU transfers on CUDA
)

On macOS the default context is already spawn (Python ≥ 3.8), so no override is needed.

5.6 Resume: Skip Already-Processed Samples

Pass skip_jsonl to remove already-processed samples from the index before any waveform data is read from HDF5:

dataset = HDF5WaveformDataset(
    h5_file="data/hdf5/continuous_waveform_usa_*.h5",
    mode="three",
    skip_jsonl="data/picks/output.jsonl",
    skip_record_type="phase_pick",
)
print(f"Original samples : {dataset.original_index_size}")
print(f"Already done     : {dataset.filtered_index_size}")
print(f"Remaining        : {len(dataset)}")

At startup the JSONL is split into byte-range chunks and parsed in parallel with ThreadPoolExecutor. Each finished record's sample key is reconstructed directly from its fields (h5_file, year_id, day_id, station_id, channel_family, channels). The matching index entries are dropped before any __getitem__ call.

5.7 Output Sample Dictionary

Each item returned by __getitem__ (or delivered by the DataLoader after waveform_collate_fn) is a plain Python dict:

{
    # Identity
    "mode": "three",
    "h5_file": "data/hdf5/continuous_waveform_usa_20190701.h5",
    "year_id": "2019-01-01T00:00:00.000000Z",
    "day_id":  "2019-07-01T00:00:00.000000Z",
    "station_id": "BK.BDM.00",

    # Channel layout
    "channel_family": "BH",
    "channels": ["BHE", "BHN", "BHZ"],   # one entry per component slot
    "component_order": "E/N/Z or 1/2/3",
    "is_z_only": False,
    "z_only_replicated": False,

    # Waveform
    "waveform": torch.Tensor,            # shape [T, 3], dtype float32
    "sampling_rate": 100.0,              # Hz after optional resampling
    "original_sampling_rate": 40.0,
    "target_sampling_rate": 100.0,
    "resampled": True,
    "npts": 8640001,

    # Timing
    "starttime": "2019-07-01T00:00:00.000000Z",
    "endtime":   "2019-07-01T23:59:59.990000Z",

    # Station metadata
    "station_info": {
        "station_id": "BK.BDM.00",
        "network": "BK",
        "station": "BDM",
        "location": "00",
        "longitude": -122.2486,
        "latitude":   37.8762,
        "elevation":  276.0,
        "location_available": True,
        "position_match_mode": "strict_time_matched_network_station_only",
        "position_is_fallback": False,
        "position_history": [...],
        ...
    },

    # Per-channel metadata (keyed by channel code)
    "channel_info": {
        "BHE": {"channel": "BHE", "segment_count": 3, "starttime": "...", ...},
        "BHN": {...},
        "BHZ": {...},
    },

    # Raw segment metadata (empty list when include_segments_metadata=False)
    "segments": [],
}

5.8 Minimal Example Script

scripts/example_dataloader.py demonstrates the full pipeline in under 90 lines:

python scripts/example_dataloader.py \
  --h5_input data/hdf5/continuous_waveform_usa_20190701.h5 \
  --n_samples 3

Sample output:

── Sample 1 ──────────────────────────────────────────
  station_id    : BK.BDM.00
  network       : BK.BDM
  channels      : ['BHE', 'BHN', 'BHZ']
  starttime     : 2019-07-01T00:00:00.000000Z
  sampling_rate : 100.0 Hz
  waveform shape: (8640001, 3)  (86400.0 s  ×  3 components)
  waveform dtype: torch.float32
  Z-only        : False
  location      : lon=-122.2486  lat=37.8762
  ch[E/1]  min=-1.234e-05  max=+2.345e-05  std=3.210e-06
  ch[N/2]  min=-1.100e-05  max=+1.987e-05  std=2.890e-06
  ch[Z/3]  min=-9.870e-06  max=+1.543e-05  std=2.654e-06

6. Recommended Reporting Quantities

Recommended reporting quantities for reusable continuous-monitoring studies:

• —Declared coverage level (C0--C3), NSLC families, window, component, sampling, response, and gap requirements.
• —Coverage-aware recall: the label-centred fraction of coverage-qualified reference arrivals matched within the declared tolerance.
• —Pick-centred matched fraction: the fraction of eligible automatic picks used by the one-to-one assignment within that tolerance.
• —Total automatic-pick volume and catalogue-unmatched pick volume per unit time; unmatched picks are not presumed to be false detections.
• —Thresholds, preprocessing, channel selection, and phase normalization.
• —Residual summaries when needed for timing diagnostics.

7. Data-Product Rationale

SeismicX-Cont is organized as a reusable continuous-monitoring data product with separate reporting views for:

• —high-event-density conditions, where waveform coverage and reference-label denominators must be documented carefully;
• —lower-event-rate background conditions, where catalog-unmatched automatic picks, station coverage, and metadata completeness become prominent.

The structure supports reproducible reporting and workflow diagnostics without implying a standardized or fair comparison among independently trained picking systems.

8. Summary

SeismicX-Cont provides a processed continuous waveform database, coverage metadata, CEED-derived annotations, instrument-response metadata, validation scripts, and reusable access utilities.

It is well suited for continuous seismic detection studies, monitoring-workflow diagnostics, phase-association experiments, and large-scale waveform analysis where provenance, coverage, and source-provider attribution must remain auditable.

9. Waveform Index API and Data Query (with macOS External Drive Notes)

This section introduces the SQLite-based waveform indexing and querying system.

9.1 Core API

from utils.waveform_index_api import query_and_merge

merged, rows = query_and_merge(
    db_file="data/index/waveform_index.sqlite",
    network="BK",
    station="BDM",
    location="*",
    channel="BH*",
    starttime="2019-07-01T00:00:00",
    endtime="2019-07-01T01:00:00",
    fill_value=0.0,
)

merged is a dict keyed by "network.station.location.channel", each value containing data (ndarray), sampling_rate, starttime, endtime, filled_ratio, and segments. The same API can apply the released instrument-response JSON before returning arrays:

merged, rows = query_and_merge(
    db_file="data/index/waveform_index.sqlite",
    network="BK",
    station="BDM",
    channel="BHZ",
    starttime="2019-07-01T00:00:00",
    endtime="2019-07-01T00:01:00",
    instrument_response_json="data/response/instrument_responses.json",
    remove_instrument_response=True,
    response_output="VEL",
    response_pre_filt=(0.2, 0.5, 20.0, 45.0),
    response_water_level=None,
)

When response processing is enabled, each returned waveform item includes an instrument_processing metadata dict with the matched native response_id, response epoch, optional simulated target response id, processing flags, and any error message.

9.2 Wildcard Query Mechanism

* is converted to SQL LIKE (%). Supported: * (match all), BK (exact), B* (prefix), *Z (suffix), BH* (channel family).

9.3 Build SQLite Index

python utils/hdf5_waveform_index.py build \
  --h5 "data/hdf5/continuous_waveform_usa_*.h5" \
  --db data/index/waveform_index.sqlite

On external drives, prefer the wrapper below. It builds the SQLite database on local temporary storage first and then copies the finished index into data/index/:

./scripts/build_waveform_index.sh

9.4 Example Script

python scripts/example_query_waveform.py \
  --db data/index/waveform_index.sqlite \
  --network BK --station BDM \
  --starttime 2019-07-01T00:00:00 \
  --endtime 2019-07-01T01:00:00

Add response removal or target-response simulation when amplitude-aware access is needed:

python scripts/example_query_waveform.py \
  --db data/index/waveform_index.sqlite \
  --network BK --station BDM --channel BHZ \
  --starttime 2019-07-01T00:00:00 \
  --endtime 2019-07-01T00:01:00 \
  --remove_response \
  --response_pre_filt 0.2 0.5 20 45 \
  --response_water_level none

9.5 ⚠️ macOS External Drive Issue

SQLite requires file locking and journal/WAL files. External drives (exFAT, NTFS, network drives) often lack reliable locking support, causing sqlite3.OperationalError: attempt to write a readonly database even when HDF5 writes succeed normally.

Solution (recommended): keep the SQLite index on a local disk while HDF5 data stays on the external drive:

# HDF5 on the T5 project drive
--h5 /Volumes/T5/publish/data/hdf5/*.h5

# SQLite on local disk
--db ~/waveform_index.sqlite

Test SQLite writability explicitly:

python -c "import sqlite3; sqlite3.connect('/tmp/seismicx_sqlite_test.db').execute('CREATE TABLE t(a)')"

10. Phase Picking: scripts/run_picker_to_jsonl.py

Runs a TorchScript seismic phase picker over every sample in the DataLoader and writes results to a JSONL file. Supports Mac/MPS, CUDA (single or multi-GPU), and CPU with automatic device selection, periodic model reloading, and built-in resume.

10.1 Quick Start

Mac / Apple Silicon (MPS):

python scripts/run_picker_to_jsonl.py \
  --h5_input 'data/hdf5/continuous_waveform_usa_*.h5' \
  --output_jsonl data/picks/output.jsonl \
  --picker_model pickers/pnsn.v1.jit \
  --polar_model pickers/polar.onnx \
  --device mps

CUDA with multi-process prefetch:

python scripts/run_picker_to_jsonl.py \
  --h5_input 'data/hdf5/continuous_waveform_usa_*.h5' \
  --output_jsonl data/picks/output.jsonl \
  --picker_model pickers/pnsn.v1.jit \
  --polar_model pickers/polar.onnx \
  --device cuda \
  --batch_size 4 \
  --num_workers 4 \
  --prefetch_factor 2

Resume an interrupted run (default behavior):

Simply re-run the same command. The script scans the existing JSONL at startup, rebuilds the set of already-processed sample keys, and removes them from the DataLoader index before any waveform data is read.

10.2 CLI Arguments

10.3 Device-Specific Notes

MPS (Mac / Apple Silicon)

Use --num_workers 0 (single-process). The MPS TorchScript allocator accumulates internal Metal buffer state with each sess(xt) call; torch.mps.empty_cache() does not fully reclaim it. The model is reloaded every 50 samples by default (--reload_model_interval -1) to reset allocator state. Increase the interval for very short waveforms if the reload overhead is noticeable; set --reload_model_interval 0 to disable.

For long runs (thousands of samples), even model reloads cannot fully reclaim the Metal allocator's process-level state. The recommended pattern is to let the OS reclaim memory on each process exit using --max_samples and a restart loop:

while true; do
  python scripts/run_picker_to_jsonl.py \
    --h5_input 'data/hdf5/continuous_waveform_usa_*.h5' \
    --output_jsonl data/picks/output.jsonl \
    --picker_model pickers/pnsn.v1.jit \
    --polar_model pickers/polar.onnx \
    --device mps --max_samples 200
  [ $? -ne 75 ] && break
done

The script exits with code 75 when --max_samples is reached and code 0 when all samples are finished. --resume is enabled by default, so each restart skips already-written samples at the dataset level (no waveform data is re-read from HDF5).

CUDA

Use --num_workers 4 or more to overlap HDF5 I/O with GPU inference. The spawn context is selected automatically on Linux. CUDA allocator state does not accumulate across TorchScript calls, so model reload is disabled by default.

CPU

Single-process; no special memory management needed.

10.4 Resume Mechanism

At startup the script passes skip_jsonl=output_jsonl to HDF5WaveformDataset. The JSONL is scanned in parallel using ThreadPoolExecutor. Each finished record's sample key is reconstructed from its stored fields (h5_file, year_id, day_id, station_id, channel_family, channels). Matching index entries are dropped from the DataLoader — their waveform data is never read from HDF5.

On the second and subsequent runs, startup takes O(N_finished × JSON parse cost). For very large JSONL files (>10 GB) this may take 30–120 seconds but is far cheaper than re-running inference.

10.5 Output JSONL Format

Each output line is a self-contained JSON record. Two record_type values are written:

`"phase_pick"` — one record per detected phase:

{
  "record_type": "phase_pick",
  "phase_id": 0,
  "phase_name": "Pg",
  "phase_prob": 0.987,
  "phase_sample": 15234.0,
  "phase_relative_time": 152.34,
  "phase_time": "2019-07-01T00:02:32.340000Z",
  "polarity": "U",
  "polarity_prob": 0.912,
  "polarity_available": true,
  "snr": 12.4,
  "amplitude": 3.14e-05,
  "pre_std": 2.10e-06,
  "post_std": 2.60e-05,
  "pre_abs_p95": 4.11e-06,
  "post_abs_p95": 5.87e-05,
  "channels": ["BHE", "BHN", "BHZ"],
  "channel_family": "BH",
  "component_order": "E/N/Z or 1/2/3",
  "is_z_only": false,
  "z_only_replicated": false,
  "waveform_shape": [8640001, 3],
  "sampling_rate": 100.0,
  "original_sampling_rate": 40.0,
  "target_sampling_rate": 100.0,
  "resampled": true,
  "h5_file": "data/hdf5/continuous_waveform_usa_20190701.h5",
  "year_id": "2019-01-01T00:00:00.000000Z",
  "day_id":  "2019-07-01T00:00:00.000000Z",
  "station_info": {
    "station_id": "BK.BDM.00",
    "network": "BK",
    "station": "BDM",
    "location": "00",
    "longitude": -122.2486,
    "latitude":   37.8762,
    "elevation":  276.0,
    "location_available": true,
    "position_in_time_range": true,
    "position_is_fallback": false
  }
}

`"error"` — one record per sample where inference failed:

{
  "record_type": "error",
  "station_id": "BK.BDM.00",
  "h5_file": "...",
  "year_id": "...",
  "day_id": "...",
  "sample_key": "...|...|...|...|BH|BHE,BHN,BHZ",
  "error": "RuntimeError: ..."
}

Phase ID → name mapping:

11. Evaluating Picks: essd_scripts/evaluate_picks.py

Matches automatically generated phase picks against provenance-preserving reference annotations, which include separate manual and operational automatic subsets. It uses a SQLite index to handle JSONL output files of 40 GB or more without loading them into memory. The optional matching records under data/validation/multimodel_phase_matching/ are usability and interoperability checks for the data product and reporting scripts. They are not relative model- assessment evidence; their large source automatic-pick JSONL files are not part of the versioned extension.

11.1 Workflow

Step 1 — Build the SQLite index (first run only):

python essd_scripts/evaluate_picks.py \
  --auto-jsonl  data/picks/output.jsonl \
  --index-db    ~/output.index.sqlite \
  --build-index \
  --drop-existing

This streams through the JSONL, batch-inserts picks into SQLite (50,000 rows per commit), and builds indices on (station_id, phase_name, sec_key) and time_epoch for fast lookup. The index file is typically 3–5× smaller than the JSONL. Use --drop-existing when rebuilding an index after updating the source JSONL, so old picks are not retained.

Step 2 — Evaluate recall and residuals:

python essd_scripts/evaluate_picks.py \
  --auto-jsonl  data/picks/output.jsonl \
  --label-json  data/label/annotations_for_continuous_hdf5.json \
  --reference-db data/label/reference_arrivals.sqlite \
  --index-db    ~/output.index.sqlite \
  --outdir      eval_picks/eval_phasenet \
  --tp-tol      1.5 \
  --err-window  5.0

Step 3 — Add an optional catalogue-conditioned reference-window output diagnostic:

python essd_scripts/evaluate_picks.py \
  --auto-jsonl  data/picks/output.jsonl \
  --label-json  data/label/annotations_for_continuous_hdf5.json \
  --reference-db data/label/reference_arrivals.sqlite \
  --index-db    ~/output.index.sqlite \
  --outdir      eval_picks/eval_phasenet \
  --tp-tol      1.5 \
  --coverage-level C0 \
  --output-eligibility-db data/label/output_eligibility.sqlite \
  --output-name phasenet \
  --reference-window-s 5 \
  --reference-window-s 30

Each --reference-window-s value is a half-width T and must be at least the one-to-one matching tolerance epsilon = --tp-tol. The utility applies the same declared C0--C3 configuration used for recall, retains references inside the selected output's processing domain, merges +/-T windows by exact station-location and phase, and intersects them with that processing domain. It reports the catalogue-conditioned match fraction together with raw and effective processed-window durations, eligible reference count, matched count, and catalogue-unmatched count. This is a precision-oriented output diagnostic, not an estimator of conventional full-stream precision. Both T and epsilon, the coverage level, and the output eligibility domain must be reported.

Step 4 — Generate figures (optional, requires matplotlib + scipy):

python essd_scripts/evaluate_picks.py \
  --auto-jsonl  data/picks/output.jsonl \
  --label-json  data/label/annotations_for_continuous_hdf5.json \
  --reference-db data/label/reference_arrivals.sqlite \
  --index-db    ~/output.index.sqlite \
  --outdir      eval_picks/eval_phasenet \
  --plot

Inspect index statistics:

python essd_scripts/evaluate_picks.py \
  --index-db ~/output.index.sqlite \
  --db-info

11.2 CLI Arguments

11.3 Catalogue-conditioned reference-window match fraction

For exact station-location s and normalized phase phi, the utility first selects manual references satisfying the declared C0--C3 coverage configuration and lying inside the assessed output's eligibility intervals. It merges [t_r-T, t_r+T] windows and intersects the union with those same eligibility intervals. The numerator is the number of deterministic one-to-one matches within epsilon; the denominator is every automatic pick in the resulting effective processed-window domain. Other denominator picks are catalogue- unmatched, not confirmed false detections.

This fraction is strongly reference-conditioned when T is small. As T increases, the union covers more continuous output, but catalogue incompleteness becomes more consequential. It is therefore an output-audit diagnostic rather than conventional precision. Reuse requires the same coverage, waveform, reference, and workflow processing domains, preprocessing, threshold, phase mapping, T, epsilon, and one-to-one matching rule. Report both raw reference-window duration and effective processed-window duration.

11.4 Phase Map

The --phase-map argument controls how label phase names are matched to automatic phase names. The default normalization is:

P  → P, Pg, Pn
S  → S, Sg, Sn
Pg → Pg
Sg → Sg
Pn → Pn
Sn → Sn

Assignments are one-to-one within station, normalized phase, and label subset; one automatic-pick row cannot satisfy two reference arrivals. The time-ordered assignment maximizes matches within the declared TP tolerance before considering wider diagnostic-window assignments and total absolute residual.

Override with a semicolon-separated list:

# Match label P to Pg or Pn; label S to Sg or Sn
--phase-map "P:Pg,Pn;S:Sg,Sn"

# Strict: label P must match Pg only
--phase-map "P:Pg;S:Sg"

11.5 Annotation JSON Format

The label file follows a nested structure:

years → days → events → stations → picks[]

Each pick entry must have at minimum:

{
  "phase": "P",
  "time":  "2019-07-01T00:02:32.340000Z",
  "status": "manual",
  "station_id": "BK.BDM.00"
}

Optional fields: network, station, location, event_id, distance_km.

11.6 Output Files

All outputs are written to --outdir:

Subsets reported: all (every label pick), manual (manually reviewed only), automatic (automatically labeled only).

`summary.json` structure (excerpt):

{
  "tp_tolerance_s": 1.5,
  "residual_window_s": 5.0,
  "label_phase_count_all_status": {"P": 1240, "S": 1198},
  "subsets": {
    "all": {
      "P": {
        "n_label": 1240,
        "n_matched_within_tp_tol": 1102,
        "recall": 0.889,
        "residual_mean_s": 0.031,
        "residual_std_s": 0.214,
        "residual_median_s": 0.018,
        "residual_abs_p90_s": 0.45,
        "gaussian_fit_all_within_err_window": {"mean": 0.031, "std_mle": 0.213, "std_unbiased": 0.213},
        "student_t_fit_all_within_err_window": {"df": 4.2, "loc": 0.029, "scale": 0.18}
      }
    }
  },
  "auto_pick_count": {
    "total": 458320,
    "by_auto_phase": {"Pg": 231044, "Sg": 198210, "Pn": 18700, "Sn": 10366},
    "mapped_to_label_phase": {"P": 231044, "S": 198210}
  },
  "reference_window_match": [{
    "window_half_width_s": 5.0,
    "matching_tolerance_epsilon_s": 1.5,
    "reference_subset": "C0-covered manual arrivals",
    "coverage_level": "C0",
    "workflow_eligibility": {
      "source": "data/label/output_eligibility.sqlite",
      "output_id": 1,
      "output_name": "phasenet",
      "valid_input_required": true
    },
    "by_phase": {
      "P": {
        "n_coverage_qualified_manual_references": 1020,
        "n_workflow_eligible_manual_references": 1005,
        "n_merged_reference_windows": 810,
        "n_effective_processed_windows": 815,
        "reference_window_union_s": 9650.0,
        "effective_processed_window_s": 9400.0,
        "n_reported_picks_in_effective_windows": 1180,
        "n_matched_one_to_one": 910,
        "n_catalogue_unmatched_in_effective_windows": 270,
        "reference_window_match_fraction": 0.771,
        "reference_recovery_fraction": 0.905
      }
    }
  }]
}

The numbers in this JSON fragment are illustrative schema values, not released workflow results.

12. REAL Association: scripts/run_real_association.py

scripts/run_real_association.py connects the SeismicX-Cont picker JSONL format to the REAL association algorithm. It compiles scripts/real/real.c, normalizes picker phases to P/S, writes REAL input files, runs association-based event detection, and exports a workflow JSONL containing both time-window phase candidates and associated earthquake events. The output can be used directly for event-detection evaluation or passed to a user-supplied earthquake-location program through the associated-arrival list.

python scripts/run_real_association.py \
  --picks-jsonl data/picks/pnsn_v3_diff.phase.jsonl \
  --output-jsonl data/associated/real_events.jsonl \
  --starttime 2019-07-01T00:00:00Z \
  --endtime 2019-07-08T00:00:00Z

The output data/associated/real_events.jsonl includes one real_association_run metadata record, one real_input_pick record for each phase pick that entered REAL after filtering/NMS, and one real_event record for each associated event. Event arrivals keep a source_pick.real_input_pick_id so users can trace each associated phase back to the time-window phase record and then to the original picker JSONL line, station metadata, probability, channel family, and HDF5 file. The companion real_events.summary.json records filter counts and REAL run commands.

The default settings are conservative workflow defaults rather than calibrated catalog-production parameters. Tune --min-prob, --real-s, --real-r, and --real-g for a particular picker, period, and velocity model. Detailed usage notes are in scripts/README_real_association.md.

The association JSONL is intentionally simple. A downstream locator can ignore the REAL-specific diagnostic fields and consume only real_event.picks[], where each associated arrival gives station identity, phase name, pick time, and a link back to the original automatic-pick record. Preliminary latitude, longitude, and depth_km fields are included when the associator provides them, but they are not required for downstream location workflows.

12.1 Event-Level Catalog Comparison

Use essd_scripts/compare_associated_events.py to compare associated events against the SeismicX-Cont catalog or any event JSONL following the documented schema:

python essd_scripts/compare_associated_events.py \
  --pred-jsonl data/associated/real_events.jsonl \
  --catalog data/label/annotations_for_continuous_hdf5.json \
  --outdir eval_events/real_vs_catalog

An event is counted as catalogue matched when it matches a reference event with epicentral distance error < 20 km and origin-time error < 3 s. The script reports catalogue-matched, catalogue-unmatched, and unrecovered catalogue-event counts together with catalogue match and recovery fractions and time/location/depth/magnitude error statistics for matched events. These catalogue-relative labels do not assign physical false-event status to unmatched workflow events. The generic event JSONL format is documented in essd_scripts/README_event_comparison.md.

13. License, Citation, and Checksums

Licensing is layer-specific. See LICENSE and LICENSE_CLARIFICATION.md for the full terms.

• —Waveform sample arrays stored in the HDF5 files are processed and structured derivatives of source waveform records. Publication and redistribution of these arrays have been clarified with the relevant U.S. seismic data providers, and their use is limited to non-commercial scientific research under upstream-compatible data-use terms.
• —Except for the waveform sample arrays, all assets in the full and mini releases are provided under CC BY 4.0. This includes metadata schemas, coverage indexes, event and phase annotations, response metadata, manifests, validation summaries, automatic-pick and consensus-candidate products, figures, reuse documentation, code in scripts/, essd_scripts/, and utils/, and notebooks.

Only the waveform sample arrays are excluded from the SeismicX-Cont CC BY 4.0 grant. They are not relicensed by SeismicX-Cont and remain subject to applicable source-provider terms. Source lineage and citation requirements also remain in force for source-attributed metadata and derived records. Users must cite SeismicX-Cont together with the original SCEDC, NCEDC, CEED, and contributing network sources as appropriate.

The release version is recorded in VERSION; releases should also be tagged with the same version string, for example v1.0.0. Machine-readable citation metadata are provided in CITATION.cff.

The main-release checksum and file-size manifests can be regenerated with:

./scripts/write_manifest.sh

The script writes manifest_sha256.txt and manifest_filesizes.tsv. Both files exclude local build products, manuscript files, generated figures, mini-release files, temporary files, picker outputs, and intermediate hourly subsets. The HuggingFace and ModelScope repositories are kept aligned to these same manifest files, so either hosting service can be checked against the same paths, SHA-256 hashes, and file sizes.

ESSD manuscript tables and figures can be regenerated from released data products and evaluation outputs with:

./essd_scripts/reproduce_manuscript_outputs.sh

This command writes audit reports under essd_scripts/outputs/ and rebuilds the manuscript figures under figures/. The manuscript-number consistency step runs only when overleaf-essd/template.tex or essd/template.tex is present. A data release downloaded without the manuscript source still runs the inventory, coverage, waveform-quality, and figure checks and records the manuscript check as skipped.

Contact

SeismicX is developed by:

• —Ziye Yu - yuziye@cea-igp.ac.cn
• —Yuqi Cai - caiyuqiming@foxmail.com
• —Xin Liu - xinliu_geo@outlook.com
• —Yanru An
• —Fajun Miu
• —Tengchao Dong
• —Tingting Li - 972516012@qq.com
• —Xinghui Huang - huangxinhui@seis.ac.cn
• —Lu Li
• —Ming Ma - 925100762@qq.com

<p align="center"> _{Built with ❤️ for the seismology community} </p>