mark000071/Ship_trajectory_interpolation

EnvShip-Voyage: Long-Term Ship Trajectory Imputation (and Prediction)

A benchmark for filling in long gaps in ship trajectories. Ships broadcast their position over AIS, but the signal drops out for minutes to hours at a time. This dataset gives you clean, fixed-length windows of vessel movement with the gaps marked, so you can train and fairly compare models that reconstruct the missing part of a track.

It is built from six months of Danish AIS data (cargo and tanker vessels) and comes with the environment around each track already sampled — water depth, distance to shore, and distance to the nearest shipping fairway — which most trajectory datasets don't provide.

• —179,131 ready-to-use samples (windows), split so that no vessel appears in more than one split.
• —Two window lengths: 12 h (144 steps) and 6 h (72 steps), on a fixed 5-minute grid.
• —A frozen mask bank: for every sample, a set of pre-computed "hide these steps" patterns, so everyone evaluates on exactly the same gaps.
• —Baseline results (linear/spline interpolation) included as a reference point.

Quick facts

What's in the repo

voyage_v1/regions/dma/build/
  s9/                     # the samples
    samples/track_i/part-*.parquet    # imputation samples (one row = one window)
    samples/track_p/part-*.parquet    # same windows split for causal prediction
    sample_index.parquet
  s10/                    # the mask bank
    masks/part-*.npz                  # bit-packed hide-these-steps masks
    mask_index.parquet
  s11/sample_index_split.parquet      # train/val/test label + metadata per sample
  s12/baseline_results.json           # linear/spline/hold reference scores

voyage_v1/basemaps/dma/               # the environment layer (shared, streamed per point)
  basemap.zarr/                       # land/water, depth, distance-to-shore
  fairway_dist.zarr/                  # distance to nearest fairway
  fairway_network/                    # fairway lines + traffic-separation schemes
  basemap_meta.json

pipeline/                             # the full build code (raw AIS -> this dataset)
tools/                                # dataloader, environment sampler, metrics
docs/                                 # schema, data card, design notes
configs/voyage.yaml                   # every setting used to build the dataset

The features in each sample

Each sample is one row with list-columns (one value per time step, in time order). A 12 h sample has 144 values per column, a 6 h sample has 72.

Position and motion

• —lat, lon — WGS84 coordinates (NaN where the ship's position is unknown at that step)
• —x_m, y_m — local metres (centred on the voyage), convenient for models
• —sog_kn — speed over ground (knots)
• —cog_sin, cog_cos, head_sin, head_cos — course and heading, split into sine/cosine so they wrap around correctly

Which steps are real vs missing (this is the point of the dataset)

• —observed_mask — 1 if this step is a genuine AIS observation, 0 if it was missing
• —position_available — 1 if a position is known (observed, or filled by a short safe interpolation)
• —interp_provenance — 0 = observed, 1 = short-gap interpolation, 2 = long gap left as an imputation target

Environment along the track (sampled from the basemap)

• —is_water, depth_m — water/land and water depth
• —d_shore_m, d_nav_m — distance to shore, distance to navigable water
• —dist_fairway_m, in_fairway — distance to the nearest shipping fairway, and whether the ship is on one

Per-sample info (in s11/sample_index_split.parquet): split, kept, difficulty_tier, real_coverage, n_real_gaps, max_gap_min, path_efficiency, gc_distance_km, travelled_km, ship_class, window_len_h, mmsi_hash.

Only rows with kept == True are part of the 179,131-sample benchmark. The file also keeps the rest (capped near-duplicate windows and a temporal pretraining pool) in case you want them.

The mask bank (how to set up an imputation task)

For every sample we pre-computed several masks, stored bit-packed in s10/masks/part-*.npz under the key "<sample_id>|<mask_name>". A mask is a 0/1 array the same length as the window; 1 means "hide this step and ask the model to reconstruct it."

Evaluate only where you have ground truth: the steps that count are mask == 1 AND observed_mask == 1. Note realgap has no ground truth by definition (those steps were never observed), so use it for training or qualitative demos, not for scoring.

How to use it

import sys; sys.path.insert(0, "tools")
from dataloader import VoyageLoader

loader = VoyageLoader("voyage_v1/regions/dma/build")

for s in loader.iter_track_i(split="train", mask_name="cont_6h", window_len_h=12):
    # s.lat, s.lon            -> the trajectory (list of 144 values)
    # s.feats                 -> dict of the other per-step features (speed, course, depth, ...)
    # s.observed_mask         -> which steps are real
    # s.mask                  -> which steps to reconstruct (cont_6h here)
    # s.eval_mask             -> mask AND observed  (the steps you can actually score)
    ...

A minimal recipe: build the model input by blanking the masked positions, predict them, and score with tools/metrics.py (MAE / RMSE in metres, ADE / FDE). Compare against s12/baseline_results.json.

Environment lookups anywhere (not just along the samples) use the streaming basemap:

from env_loader import EnvLoader
env = EnvLoader("voyage_v1/basemaps/dma")
df = env.sample_track(lat_array, lon_array)   # depth, distance-to-shore, distance-to-fairway per point

How the dataset was built

The pipeline (in pipeline/) turns raw daily AIS files into this dataset. In short:

1. 1.Ingest (S1–S2, `s1s2_ingest_month.sh`). Download one month of AIS, standardise the columns, drop bad rows, sort and de-duplicate, and keep only cargo and tanker vessels.
2. 2.Assemble voyages (S3, `s3_assemble_voyages.py`). Group each vessel's points into voyages, cutting a new voyage wherever there's a gap longer than 2 hours. Keep voyages that are at least 4 hours long and mostly inside the region.
3. 3.Resample (S4, `s4_resample.py`). Put every voyage on a regular 1-minute grid, mark which steps are real, and interpolate only very short gaps (≤ 15 min). Check the motion is physically consistent.
4. 4.Environment (S5–S7). Build a shared map of depth, land/water, distance-to-shore, and fairways (s5_basemaps.py, s6_fairway.py), then sample those values along every voyage (s7_sample_env.py).
5. 5.Window (S9, `s9_window.py`). Slide 12 h and 6 h windows over the voyages on a 5-minute grid.
6. 6.Masks (S10, `s10_masks.py`). Pre-compute the frozen mask bank.
7. 7.Split and curate (S11, `s11_split.py`). Split by vessel so no vessel leaks across train/val/test, cap near-duplicate windows, and recalibrate difficulty tiers.
8. 8.Baselines, checks, packaging (S12–S14).

The exact settings live in configs/voyage.yaml. run_full_build.sh runs the whole thing; every stage is resumable.

Raw data — not included here, but here's where to get it

We deliberately do not ship the raw AIS files (they are large and already public). To rebuild from scratch, or to extend to more months/regions, download:

• —AIS (the trajectories). Danish Maritime Authority open AIS, one zip per day: http://aisdata.ais.dk/aisdk-YYYY-MM-DD.zip (this release used 2026-01 through 2026-06). Portal: <https://www.dma.dk/safety-at-sea/navigational-information/ais-data>
• —Bathymetry (depth). ETOPO 2022 15-arcsec, via NOAA CoastWatch ERDDAP (use the .nc endpoint, not the rendered GeoTIFF): <https://coastwatch.pfeg.noaa.gov/erddap/griddap/ETOPO2022v1_15s.html>
• —Fairways / traffic-separation schemes. OpenStreetMap seamark data via the Overpass API: <https://overpass-api.de/>

Point the pipeline at the downloaded AIS zips and run pipeline/run_full_build.sh.

Also not included: intermediate build products (the per-day filtered CSVs, the raw assembled voyages, the resampled tracks) — about 35 GB that the pipeline regenerates on its own. Only the finished, ready-to-use dataset is here.

Known limitations (worth reading)

• —Coarse near the coast. The depth/land map (ETOPO, ~450 m grid) is too coarse for narrow Danish channels, so roughly a quarter of near-shore points sit on a cell the map calls "land." These points are flagged, never deleted — open-water depth and distances are accurate, but treat the water/land label as approximate very close to shore.
• —Two ship types only. Cargo and tanker. Other vessel types were filtered out.
• —One region, six months. Danish waters, first half of 2026. Cross-region generalisation is future work.

License and attribution

• —This dataset (the packaged samples, masks, splits, and code) is released under CC-BY-4.0. If you use it, please cite it (below).
• —It is derived from public sources, whose terms you should also respect:
• —AIS data © Danish Maritime Authority, provided as open data.
• —Fairway / seamark data © OpenStreetMap contributors, licensed under the Open Database License (ODbL).
• —Bathymetry from ETOPO 2022 (NOAA / NCEI), public domain.

Citation

@misc{ma_envship_voyage_2026,
  title  = {EnvShip-Voyage: A Benchmark for Long-Term Ship Trajectory Imputation},
  author = {Ma, Kun},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/datasets/mark000071/Ship_trajectory_interpolation}}
}

Questions or issues: open a discussion on the dataset page.