ChenmingWu/pointcalib-corpus

PointCalib corpus — 1,051,756 frames, one format

Nine source datasets normalised into a single streamable WebDataset, plus the frozen evaluation protocol and checkpoints behind our reported numbers. The point is not to mirror upstream archives (TartanAir, Hypersim are already on the Hub) but to remove the nine-decoder / nine-resolution / nine-depth-convention tax: one format, one depth convention, streamable.

Contents

MatrixCity ships as 12 prefixes (matrixcity_{small,big}_<regime>_{test,train}) rather than one, because each was streamed independently: the upstream street split is 824 GB against ~250 GB of free disk, so every chunk was downloaded, encoded, pushed and deleted before the next began, and each needed its own resumable shard series.

Depth-range coverage differs by design: matrixcity is the far-range split (real geometry out to 250 m, 90-degree-FOV city streets), urbansyn the wide-field driving one (median frame depth 12.4 m), mvssynth the street-level one (median 27.0 m, 86.6% of pixels within 80 m), pointodyssey the indoor one (median 3.3 m) and the only source of articulated MOTION here — every other synthetic split is a static scene filmed by a moving camera.

MatrixCity: four conventions worth knowing if you use the upstream archives

All four were measured here from the data alone and only then checked against github.com/city-super/MatrixCity, which agreed on every one. In these shards they are already applied — metres, 0 for invalid, per-frame K.

• —depth is in centimetres (depth_m = value / 100). Independently visible in transforms.json: the rotation rows of rot_mat have norm 0.01, not 1, so the unit conversion is baked into the pose matrix.
• —it is planar z-depth (ground-plane residual 11.6 units under the z reading vs 25.7 euclidean).
• —65504 is the sky sentinel, not a distance — it is float16's maximum, so 655.04 m is the farthest representable value and sky saturates there. Sky is ~45% of pixels.
• —the optical axis is −z (NeRF, X-right/Y-up/Z-backward): two-view reprojection reaches 87% inliers under −z where +z peaks at 27%.

Frames whose valid (non-sky) coverage falls under 20% are omitted — 3.4% of the selection overall, but 18% in the sky-heavy small_outside_test chunk.

MVS-Synth: two conventions worth knowing if you use the upstream archives

Its EXR depth is decimetres, not metres — nothing upstream documents this, and reading it as metres is a silent 10x error that looks plausible (median frame depth 270 "m", nearest content 75 "m"). Three independent absolute-scale anchors in one frame all bracket 0.1 m/unit: pedestrian height 1.773 m, ground-plane camera height 1.626 m, hatchback roof height 1.367 m. At the right scale it is ordinary street-level driving — median frame depth 27.0 m, 86.6% of valid pixels within 80 m. The depth is planar z-depth (two-view reprojection agrees to 2% on 96.8% of pixels under the z reading, 23.5% under the euclidean one), sky is inf, and f_x varies per sequence (530.9–578.7) so intrinsics must be read per frame. In this repo all of that is already applied: metres, 0 for invalid, per-frame K.

Also in this repo: eval_protocol/ (KITTI day-disjoint 291-frame val split with per-pixel predictions from OMNI-DC, Marigold-DC, PromptDA, PriorDA, MoGe-2, DAv2 and ours), checkpoints/, recipe/ (download + preprocessing + scoring scripts).

Sample format

Each sample is three files sharing a key: .jpg (RGB, JPEG q92, no chroma subsampling), .depth.png, .meta.json (domain, K 3x3, depth_scale, h, w).

Depth is uint16 PNG at 1/256 m — the KITTI convention, not millimetres. 16-bit millimetres caps at 65.5 m, which truncates KITTI's 80 m range and the >40 m band that carries 45% of the MAE gap we measured against OMNI-DC. 1/256 m gives 255.99 m of range at 3.91 mm resolution, lossless for every domain here. 0 means invalid (sky, no return, out of range).

import io, json, tarfile
import numpy as np
from PIL import Image

with tarfile.open("tartanair-00000.tar") as t:
    key = sorted({n.split(".")[0] for n in t.getnames()})[0]
    rgb = np.asarray(Image.open(io.BytesIO(t.extractfile(f"{key}.jpg").read())))
    d16 = np.asarray(Image.open(io.BytesIO(t.extractfile(f"{key}.depth.png").read())))
    meta = json.loads(t.extractfile(f"{key}.meta.json").read())

depth = d16.astype(np.float32) / meta["depth_scale"]   # metres
valid = depth > 0                                      # 0 == invalid, do not train on it
K = np.array(meta["K"])                                # already scaled to h, w

Long side resized to 640, aspect preserved. RGB bilinear, depth NEAREST (never interpolate across a discontinuity), K scaled to match.

Read this before training on it

Shards hold consecutive frames from one environment. Shard i is frames 2000i .. 2000i+1999 in dataset order, so a shard is a contiguous trajectory segment. Without shard-level shuffling plus a large sample buffer, every batch is near-duplicate neighbouring frames and your training metrics will be optimistic.

import webdataset as wds
ds = (wds.WebDataset(urls, shardshuffle=True, nodesplitter=wds.split_by_node)
        .shuffle(5000)          # >= a few shards' worth
        .decode("pil"))

Sparse anchors are not stored. They are simulated from the dense GT at training time (LiDAR scanlines, ToF grids, uniform patterns, plus noise/outliers/mixed pixels); see recipe/config.py SensorConfig. Storing a fixed pattern would freeze one sensor model into the corpus.

Not zero-shot-clean for KITTI/VOID/ETH3D/ARKitScenes. No frame here comes from them, so they remain valid zero-shot benchmarks — but eval_protocol/ is KITTI data, for scoring only, never training.

Attribution (required)

• —TartanAir V2 — AirLab / CMU. tartanair.org: "The TartanAir V2 dataset is licensed under a Creative Commons Attribution 4.0 International License." Note the HF mirror theairlabcmu/tartanair2 tags itself bsd-3-clause; that is the license of the castacks/tartanair_tools code, not the dataset. We carry upstream CC BY 4.0.
• —Hypersim — Apple, Roberts et al., ICCV 2021. apple/ml-hypersim README: "The Hypersim Dataset is licensed under the Creative Commons Attribution-ShareAlike 3.0 Unported License." (The repo LICENSE.txt is a separate Apple software license.) The release excludes the purchased Evermotion source meshes; whether that upstream asset EULA independently permits redistribution of derived renders is not addressed by any Apple document, so that question is unresolved rather than cleared.
• —OpenScene / nuPlan — OpenDriveLab over Motional. nuScenes/nuPlan Terms of Use place the data under CC BY-NC-SA 4.0, granting the right to "reproduce and Share the Licensed Material, in whole or in part, for NonCommercial purposes only". OpenScene self-describes as "a compact redistribution of the large-scale nuPlan dataset". Third-party-supplied portions may not be redistributed without the original provider's consent, and Motional may terminate access at any time.
• —LightwheelOcc — Lightwheel AI. Upstream is CC BY-NC-ND 4.0, whose ND clause bars distributing a derivative, and re-encoded shards are a derivative. These shards are published under separate permission granted by the dataset's authors to this repository's owner. That permission does not travel with the files: if you redistribute lightwheelocc-* onward, or need terms beyond non-commercial research use of what is here, obtain your own permission from Lightwheel AI. Drop wds/lightwheelocc-*.tar if you need a corpus every part of which you may redistribute under its own public licence.
• —MatrixCity — Li et al. (ICCV 2023), CC BY-NC 4.0: attribution, non-commercial, redistribution granted. Only the street views are here; aerial is a different task and the material passes (normal / diffuse / roughness / specular / metallic) are irrelevant to depth.
• —UrbanSyn — Gómez et al. urbansyn.org licenses the dataset under CC BY-SA 4.0: attribution + share-alike, commercial use permitted. Its own json/camera_metadata.json (absent from the HF mirror) supplies the intrinsics we use; the shards carry the resulting K directly.
• —PointOdyssey — Zheng et al. (ICCV 2023), MIT licensed. The only component of this corpus whose redistribution terms are unambiguous.
• —MVS-Synth — Huang et al. (DeepMVS, CVPR 2018), rendered from Grand Theft Auto V. The project page states only "The data is for research and educational use only" — no license text and no explicit grant to redistribute, and the frames derive from Take-Two/Rockstar game assets we hold no grant under either. Included here under that research/education restriction, as a downsampled derivative; the redistribution question is recorded as unresolved in MANIFEST.json rather than treated as cleared. If you need a corpus with clean redistribution terms throughout, drop wds/mvssynth-*.tar — nothing else depends on it.
• —NYU Depth v2 — Silberman et al. Upstream publishes no license text.
• —KITTI (eval_protocol/) — Geiger et al., CC BY-NC-SA 3.0: attribution, non-commercial, share-alike.

Because NC and ShareAlike components are mixed, treat the aggregate as non-commercial and share-alike, and because of MVS-Synth, as research/education-only unless you drop that split. The aggregate is also not freely redistributable as a whole: lightwheelocc-* is here by a permission granted to this repository specifically, which does not extend to onward redistribution by you. Per-component terms above are what actually govern.

Every split is independent — wds/<prefix>-*.tar can be dropped without affecting any other, so a corpus meeting stricter terms is a matter of excluding prefixes.

SUN RGB-D is deliberately absent. rgbd.cs.princeton.edu states no license, terms, copyright or redistribution text of any kind (verified 2026-08); the only stated obligation is citation. Silence is not a grant. It was never used in any of our training runs, so excluding it costs the corpus nothing.

Measured results this corpus produced

Zero-shot KITTI, day-disjoint 291-frame val split, identical frames and masks for every method, paired Wilcoxon + 1e4 bootstrap:

PointCalib ties on RMSE and wins edge-structure preservation; it loses MAE and absrel. recipe/PLAN_32GPU_SOTA.md documents the oracle upper bounds that localise why, the 23 approaches that failed with mechanisms, and what a 32-GPU run would have to change. Current feed-forward SOTA is LDCM (KITTI 1.911 / 0.537 / 0.026 on its 4-density protocol), not OMNI-DC.

Rebuilding from upstream

uv run python recipe/download_ext.py                     # tartanair2, hypersim
uv run python recipe/expand_tartanair.py --budget-gb 700 # all 74 environments
uv run python recipe/build_webdataset.py --repo <you>/<name> \
    --domains tartanair,hypersim,nyu_labeled,openscene