izhleba/hypersim-frustum-completion

Hypersim Frustum Point Completion

A large-scale indoor point-cloud completion dataset derived from Hypersim. Each record simulates a partially observed room: one real camera view provides context, while a synthetic nearby “missing camera” frustum hides part of the scene. Models are trained to infer the masked region from visible points.

Why this dataset exists. Real 3D capture — whether from depth sensors, multi-view reconstruction, or neural fields — routinely produces incomplete geometry: occlusions, limited baselines, and out-of-frustum regions leave holes. Supervised completion methods need paired examples of what was seen and what was missing, with ground-truth geometry and appearance. Hypersim provides photorealistic indoor scenes with dense per-pixel 3D position, RGB, and normals; this release turns those renders into ~57k completion tuples with explicit visible/masked splits, signed-distance supervision to the missing region, and both camera poses.

Dataset at a glance

Layout

Scene directories sit at the repository root (one folder per Hypersim scene):

ai_002_001/
├── manifest.csv          # one row per record (split, mask stats, …)
├── stats.json            # per-scene aggregates
└── records/
    ├── ai_002_001__f0000_v00.npz
    └── ...
repro/                    # scripts + loader to rebuild from Hypersim
├── configs/hypersim_all_scenes.txt
├── scripts/
├── prob3d/
└── requirements.txt

Record schema (.npz)

Splits (per scene, frame-index based): test every 10th frame; val at frames 5, 15, 25, …; remaining frames are train.

Quick start (PyTorch loader)

cd repro
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
export PYTHONPATH=$(pwd)

from pathlib import Path
from prob3d.training.dataset import FrustumCompletionDataset

# Point at the dataset root (parent of scene folders)
root = Path("..")  # repo root when cwd is repro/
ds = FrustumCompletionDataset(root, split="train")
sample = ds[0]
print(sample["xyz"].shape, sample["rgb"].shape, sample["is_visible"].sum())

Visible points are the model input; masked points (~is_visible) are the completion target.

Held-out scenes

For strict train/val isolation, ten scenes are recommended as a physical holdout (repro/docs/HOLDOUT_SCENES.json, seed 0). Move them out of the training root before training:

cd repro
PYTHONPATH=$(pwd) python scripts/make_holdout_split.py \
    --dataset-root .. \
    --holdout-dir ../holdout_10scenes \
    --n 10 --seed 0

Reproduce from Hypersim

The full pipeline downloads public Hypersim scene zips from Apple's CDN, extracts geometry, and builds records:

cd repro
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

bash scripts/prepare_full_dataset.sh \
    --out-dir /path/to/completion_dataset \
    --workers 4

Single-scene smoke test:

PYTHONPATH=$(pwd) python scripts/prepare_completion_dataset.py \
    --scene-dir /path/to/raw/hypersim/ai_002_001 \
    --out-dir /tmp/ai_002_001 \
    --with-normals

Validate one prepared scene:

PYTHONPATH=$(pwd) python scripts/validate_completion_scene.py /tmp/ai_002_001

Generation parameters (full release)

Attribution

This dataset is a derivative work of the Hypersim Dataset (Roberts et al., Apple), licensed under CC-BY-SA 3.0.

Please cite Hypersim:

@inproceedings{roberts:2021,
  author    = {Mike Roberts and Jason Ramapuram and
               Artur Szlam and Gabriel Synnaeve and
               Soumith Chintala and Pushmeet Kohli},
  title     = {{Hypersim}: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding},
  booktitle = {ICCV},
  year      = {2021}
}

Scene assets originate from Evermotion Archinteriors volumes (see the Hypersim paper and repository for details).

Suggested Hugging Face metadata

Use these when creating the dataset repository (some are already in the YAML header above):

• —Title: Hypersim Frustum Point Completion
• —Description: Indoor point-cloud completion tuples from Hypersim with xyz, RGB, normals, and synthetic missing-camera masks. ~57k records across 457 scenes for training models that inpaint occluded 3D regions.
• —License: cc-by-sa-3.0
• —Tags: hypersim, point-cloud, 3d-completion, indoor-scenes, normals