ZhengGuangze/Stereo4D_vlbm

Stereo4D (converted to VLBM format)

This dataset contains 4,687 sequences from the Stereo4D dataset converted to the VLBM-compatible format using preprocess_stereo4d.py. The sequences have been compressed into .tar.gz archives in chunks of 50 sequences per archive.

Dataset Description

• —Source: Stereo4D (converted)
• —Format: VLBM-compatible per-sequence layout
• —Contents: RGB images, sparse depth maps (from 3D trajectories), 2D/3D trajectories, camera intrinsics and extrinsics, visibilities, and scene metadata

Scale

Dataset Structure

Each sequence directory follows this layout:

<seq_name>/
├── rgbs/
│   ├── rgb_00000.jpg
│   ├── rgb_00001.jpg
│   └── ...
├── depths/
│   ├── depth_00000.npz
│   ├── depth_00001.npz
│   └── ...
├── intrinsics.npy
├── extrinsics.npy
├── trajs_2d.npy
├── trajs_3d.npy
├── visibilities.npy
└── scene_info.json

File Descriptions

• —rgbs/: RGB frames from the left rectified video saved as JPEG (rgb_XXXXX.jpg). Resolution is 512x512 pixels.
• —depths/: Sparse depth maps saved as compressed NumPy archives (depth_XXXXX.npz). Each archive stores a float16 array under the key depth of shape (H, W) in meters. Depth is computed from 3D trajectory points projected into camera space — values are 0 where no trajectory point is observed.
• —intrinsics.npy: Camera intrinsic matrices for each frame (T, 3, 3) float16. Computed from a 60° horizontal field of view.
• —extrinsics.npy: World-to-camera extrinsic matrices (W2C) for each frame (T, 4, 4) float16. Computed as the inverse of the camera-to-world matrices provided in the source data.
• —trajs_2d.npy: 2D trajectories (T, N, 2) float16 -- pixel coordinates (x, y). Projected from 3D world coordinates using extrinsics and intrinsics.
• —trajs_3d.npy: 3D trajectories (T, N, 3) float16 -- world-space coordinates (x, y, z); zero-filled where invisible.
• —visibilities.npy: Visibility flags (T, N) float16 (1.0 visible, 0.0 not visible).
• —scene_info.json: JSON file with per-sequence metadata including num_frames, image_size, num_trajectories, source, depth_range, depth_type, and original_sequence.

Conversion Details

Stereo4D provides sparse track annotations in a compact format:

• —track_lengths: Number of frames each track appears in
• —track_indices: Frame indices for each observation
• —track_coordinates: 3D world coordinates for each observation
• —camera2world: Camera-to-world transformation matrices

The conversion expands these sparse tracks into dense (T, N, 3) arrays, projects to 2D, and computes sparse depth maps from the visible 3D points.

Data Specifications

• —Image format: JPEG (RGB), 512x512 px
• —Depth format: NPZ (float16), sparse (from tracked 3D points, 0 = unknown)
• —Annotation format: Individual .npy files (float16)
• —Extrinsics: World-to-camera (W2C) 4x4 matrices
• —Horizontal FOV: 60° (assumed)

Usage Example (Python)

import numpy as np
from PIL import Image
from pathlib import Path
import json

seq_dir = Path("data/stereo4d_vlbm/-5JaYFNtYlM_115181849")

# Load annotations
trajs_2d    = np.load(seq_dir / "trajs_2d.npy")     # (T, N, 2)
trajs_3d    = np.load(seq_dir / "trajs_3d.npy")     # (T, N, 3)
vis         = np.load(seq_dir / "visibilities.npy")  # (T, N)
intrinsics  = np.load(seq_dir / "intrinsics.npy")    # (T, 3, 3)
extrinsics  = np.load(seq_dir / "extrinsics.npy")    # (T, 4, 4)

# Load an image and depth map
frame_idx = 0
rgb = Image.open(seq_dir / "rgbs" / f"rgb_{frame_idx:05d}.jpg")
depth_npz = np.load(seq_dir / "depths" / f"depth_{frame_idx:05d}.npz")
depth = depth_npz['depth']  # float16 array (H, W), 0 = no observation

# Load scene info
with open(seq_dir / "scene_info.json", 'r') as f:
    scene_info = json.load(f)

print(scene_info)

Citation

Please cite the original Stereo4D dataset when using the converted data:

@article{chen2024stereo4d,
  title={Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos},
  author={Chen, Linyi and Herrmann, Charles and Sun, Deqing and Jampani, Varun and Yang, Ming-Hsuan and Fleet, David J. and Rubinstein, Michael and Dekel, Tali and Barron, Jonathan T.},
  year={2024}
}