Datasetpublic

SeonghuJeon/mimicgen-aligned-gt-depth

MimicGen Aligned GT Depth Sidecars Per-frame ground-truth simulator depth for the MimicGen demonstration set, aligned to the original RGB demo frames. Generated by replaying each demo's HDF5 simulator states with robosuite + MuJoCo EGL rendering. Producer: scripts/export_mimicgen_aligned_gt_depth.py in the 3DA_unified training repo. Original MimicGen data: https://mimicgen.github.io/ (CC BY 4.0). Layout 26 tasks x ~1000 demos = ~26000 NPZ files, all at the repo… See the full description on the dataset page: https://huggingface.co/datasets/SeonghuJeon/mimicgen-aligned-gt-depth.

sourceHugging Facecc-by-4.0updated 5mo agoView on Hugging Face

0likes2kdownloads

Dataset Card

MimicGen Aligned GT Depth Sidecars

Per-frame ground-truth simulator depth for the MimicGen demonstration set, aligned to the original RGB demo frames. Generated by replaying each demo's HDF5 simulator states with robosuite + MuJoCo EGL rendering.

Producer: scripts/export_mimicgen_aligned_gt_depth.py in the 3DA_unified training repo. Original MimicGen data: https://mimicgen.github.io/ (CC BY 4.0).

Layout

26 tasks x ~1000 demos = ~26000 NPZ files, all at the repo root. File names follow {task}__demo_{N}.npz.

Tasks (3 difficulty levels per family where applicable): coffee_d0/d1/d2, coffee_preparation_d0/d1, hammer_cleanup_d0/d1, kitchen_d0/d1, mug_cleanup_d0/d1, nut_assembly_d0, pick_place_d0, square_d0/d1/d2, stack_d0/d1, stack_three_d0/d1, threading_d0/d1/d2, three_piece_assembly_d0/d1/d2.

NPZ schema

Per file {task}__demo_{N}.npz (mirrors the LIBERO aligned GT depth schema):

key	shape	dtype	meaning
`depth_meters`	`(T, V=2, 256, 256)`	float32	metric depth (meters)
`frame_indices`	`(T,)`	int64	absolute frame index in the source HDF5 demo
`camera_names`	`(V,)`	object	`["agentview", "robot0_eye_in_hand"]`
`camera_intrinsics`	`(T, V, 3, 3)`	float32	OpenCV intrinsics per frame
`camera_extrinsics_c2w`	`(T, V, 4, 4)`	float32	camera-to-world poses per frame

NPZ files are deflate-compressed (np.savez_compressed).

Download

bash

pip install -U "huggingface_hub[cli]" hf_transfer
export HF_HUB_ENABLE_HF_TRANSFER=1

# Whole dataset (~1.8 TB on disk).
huggingface-cli download SeonghuJeon/mimicgen-aligned-gt-depth \
  --repo-type dataset --local-dir ./mimicgen_aligned

# Single task (1000 NPZs at once with a glob).
huggingface-cli download SeonghuJeon/mimicgen-aligned-gt-depth \
  --repo-type dataset --local-dir ./mimicgen_aligned \
  --include "coffee_d0__demo_*.npz"

Use in 3DA_unified

Point the dataset config at the downloaded directory:

yaml

datasets:
  - name: mimicgen_gtdepth
    spec:
      type: mimicgen
      gt_depth_root: <path>/mimicgen_aligned
      gt_depth_key: depth_meters
      gt_depth_scale_mode: pointmap
      gt_depth_require_geometry: true
      gt_depth_require_sidecar_file: true

The training-time loader is MimicGenDataset._load_gt_depth_targets in src/robot/dataset.py. train_robot.py routes MimicGen samples through da3_style_depth_loss and any non-GT (e.g. OxE) samples in the same batch through teacher-decoded depth MSE.