Datasetpublic

helioom/3dvlm-hypersim

3DVLM Hypersim — §6 depth format Hypersim (Apple, ICCV 2021) reconverted to the 3DVLM project's unified §6 on-disk format. 457 scenes, ~77,400 frames, V-Ray ground truth. License: CC BY-SA 3.0, same as the source. If you use this, please cite Roberts et al. (Hypersim) and apply share-alike terms to any derivatives. Layout One uncompressed .tar per scene at the repo root: hypersim/ ai_001_001.tar ai_001_002.tar ... ai_055_010.tar # 457 tars, ~620 MB… See the full description on the dataset page: https://huggingface.co/datasets/helioom/3dvlm-hypersim.

sourceHugging Facecc-by-sa-3.0updated 4mo agoView on Hugging Face

0likes2.5kdownloads

Dataset Card

3DVLM Hypersim — §6 depth format

Hypersim (Apple, ICCV 2021) reconverted to the 3DVLM project's unified §6 on-disk format. 457 scenes, ~77,400 frames, V-Ray ground truth.

License: CC BY-SA 3.0, same as the source. If you use this, please cite Roberts et al. (Hypersim) and apply share-alike terms to any derivatives.

Layout

One uncompressed .tar per scene at the repo root:

hypersim/
  ai_001_001.tar
  ai_001_002.tar
  ...
  ai_055_010.tar      # 457 tars, ~620 MB each, ~303 GB total

Each tar extracts to a {scene_id}/ directory in §6 format:

ai_001_001/
  images/
    {frame_id}.jpg       # RGB, 1024 × 768
  depth.npy              # float32, (N, H, W),  z-depth, metres
  intrinsics.npy         # float32, (N, 3, 3)
  extrinsics.npy         # float32, (N, 4, 4),  w2c (OpenCV)
  valid_mask.npy         # bool,    (N, H, W),  True = valid pixel
  meta.json

meta.json:

json

{
  "scene_id": "hypersim/ai_001_001",
  "dataset": "hypersim",
  "is_pseudo": false,
  "frame_ids": ["cam_00_frame_0001", "..."],
  "image_size": [768, 1024]
}

Conventions:

—depth is z-depth (along camera z-axis), not Euclidean. Hypersim's native Euclidean depth has already been cosine-corrected during conversion.
—extrinsics is w2c OpenCV (x-right, y-down, z-forward). Original OpenGL c2w pose was inverted and rotation columns 1, 2 flipped.
—valid_mask is honest: False for NaN, out-of-range, sky, glass.
—frame_ids order matches the first dim of every .npy. The file images/{frame_ids[i]}.jpg aligns strictly with index i.
—Raymap is not persisted; derive from (intrinsics, extrinsics) model-side (DA3 convention: unnormalised direction + z-depth + w2c).

Fixed intrinsics

Scene-constant; still stored per-frame for uniformity.

fx = fy = 886.81
cx = 511.5
cy = 383.5
H, W = 768, 1024

Download + extract

Full dataset:

bash

hf auth login
hf download helioom/3dvlm-hypersim --repo-type dataset --local-dir ./hypersim_raw
mkdir -p hypersim
for t in hypersim_raw/hypersim/*.tar; do tar -xf "$t" -C hypersim/; done

A single scene:

bash

hf download helioom/3dvlm-hypersim --repo-type dataset \
    --include "hypersim/ai_001_001.tar" --local-dir ./hypersim_raw
tar -xf hypersim_raw/hypersim/ai_001_001.tar -C ./

Minimal loader

python

import json, numpy as np, torch
from pathlib import Path
from PIL import Image

class SceneFrames(torch.utils.data.Dataset):
    """One scene → N frames. Returns (rgb, depth, K, w2c, valid)."""

    def __init__(self, scene_dir: str):
        root = Path(scene_dir)
        self.meta  = json.loads((root / "meta.json").read_text())
        self.depth = np.load(root / "depth.npy",      mmap_mode="r")
        self.K     = np.load(root / "intrinsics.npy", mmap_mode="r")
        self.E     = np.load(root / "extrinsics.npy", mmap_mode="r")
        self.V     = np.load(root / "valid_mask.npy", mmap_mode="r")
        self.imgs  = [root / "images" / f"{fid}.jpg"
                      for fid in self.meta["frame_ids"]]

    def __len__(self) -> int:
        return len(self.imgs)

    def __getitem__(self, i: int):
        rgb = np.asarray(Image.open(self.imgs[i]).convert("RGB"))
        return {
            "rgb":   torch.from_numpy(rgb),
            "depth": torch.from_numpy(self.depth[i].copy()),
            "K":     torch.from_numpy(self.K[i].copy()),
            "w2c":   torch.from_numpy(self.E[i].copy()),
            "valid": torch.from_numpy(self.V[i].copy()),
        }

Source & citation

—Repo: https://github.com/apple/ml-hypersim
—Paper: Roberts et al., Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding, ICCV 2021.
—License: CC BY-SA 3.0 (Apple Inc.). Derivatives (this dataset included) inherit the share-alike clause.

Conversion pipeline lives in the 3DVLM project repo (scripts/convert_hypersim.py). Format spec: progress/s1_plan_data.md §6.