rooty2020/text_audio_pretraining_for_GAN

bwjhdataset — Robot 3D Point-Track Dataset (droid / hrdexdb / robocasa)

A unified multi-source robot manipulation dataset with 3D point tracks of the robot gripper and arm, calibrated multi-view RGB video, language instructions, and success labels. Three splits:

robocasa additionally ships a per-subtask re-segmentation of its episodes under `robocasa/subtask_split/` (24,504 episodes, 21 shards, ~620 GB) — see Subtask split (robocasa) below. droid additionally ships a per-subtask temporal split of its episodes under `droid/subtask_split/` (28,308 episodes, 89,312 subtasks, 12 shards, ~327 GB) — see Subtask split (droid) below.

Getting the data

Each split ships as tar shards. Files larger than HF's 50 GB limit are binary-split into *.tar.partNN pieces — reassemble with:

cat hrdexdb-4.tar.part* > hrdexdb-4.tar   # parts are in order: part01, part02, ...
tar -xf hrdexdb-4.tar

Unsplit shards extract directly: tar -xf droid-01.tar. Every shard has a *.txt / *.manifest.txt file listing its episode ids.

Sample format (identical across splits)

Each episode is one directory named by its id:

<episode_id>/
  view1.mp4, view2.mp4, ...   # RGB videos (hrdexdb: up to view23; robocasa: also ego.mp4)
  gripper.npz                 # points: (T, 256, 3) float32 — 3D gripper point tracks
  robot.npz                   # points: (T, 512, 3) float32 — 3D robot arm point tracks
  object.npz                  # points: (T, 256, 3) float32 — 3D track of a manipulated object
  object2.npz, object3.npz…   # (robocasa multi-object tasks) one file per manipulated object
  meta.json                   # calibration + labels, see below

• —All 3D points are in the `view1` camera frame (OpenCV convention, meters).
• —meta.json fields: views (per-view T_cam_world 4×4 with extrinsics_convention: "cam_from_world", and intrinsics {fx, fy, cx, cy, k1, k2, p1, p2, k3, width, height}), language_instructions, language_source, success, success_source, fps, n_frames, task_category, uuid.
• —To project points into view v at frame t: p_world = inv(T_view1[t]) @ p_v1[t], then p_v = T_v[t] @ p_world, then apply view v's intrinsics (+ distortion).
• —droid / hrdexdb: all cameras are a static rig, so T_view1 (and every T_v) is a single 4×4 and the per-frame index drops out.
• —robocasa: view1/view2 ride the mobile base. Every meta carries a boolean `camera_moves` and max_base_displacement_m. For episodes where camera_moves is true, views.view1/view2 include `T_cam_world_perframe` (shape (T,4,4), camfromworld per frame) — use it so world↔camera is consistent on every view while the base drives (the wrist ego view is natively per-frame). When camera_moves is false the single T_cam_world is exact at every frame. tools/visualize_episode.py uses the per-frame field automatically. See robocasa/base_motion_filter/base_motion_index.json for the full move classification (static / settle / moving) and per-episode t_rest.

Manipulated-object tracks (robocasa)

Each robocasa episode ships one object*.npz per manipulated object — the object(s) the robot actually picks/moves. These are selected deterministically from the simulator state: an object is included iff its body moves more than 5 cm during the episode. Receptacles you place into (pans, tupperware, plates) and distractors stay static and are excluded. Multi-object tasks (e.g. PreSoakPan, SearingMeat, PackIdenticalLunches) get object.npz, object2.npz, … one per moved object; meta.json → object lists them (cfg, category, file, max_disp_m). Fixture/turn tasks (open/close, faucet, heat) manipulate no free object and ship none.

DROID split contents

The droid shards here (droid-XX-success-lang.tar) contain only success episodes with language instructions (27,615 episodes = success ∩ language-annotated, drawn from our calibration-filtered export list), and their meta.json files are already patched with the verified instructions (merged from the official aggregated annotations and the KarlP/droid 75k release) — no post-processing needed. Each shard has a matching droid-XX-success-lang.txt episode-id list. droid-11-success-lang.tar adds the same filtering over episodes exported after the main shards.

For reference/audit, droid/droid_language_recovered.json documents the full language-recovery provenance (per-entry source, donor, n_donors, donor_agreement) — see droid_language_recovered_README.md. droid-meta-patch-success.tar is the standalone metadata patch used to produce these shards.

Base-motion filter (robocasa)

robocasa's view1/view2 cameras are mounted on the robot's mobile base, and each episode stores a single frame-0 extrinsic — stale for frames after the base drives. robocasa/base_motion_filter/ classifies every episode from its per-frame base pose (source parquets, validated against sim to 0.1 mm):

tools/robocasa_base_filter.py applies this: keeps static episodes unchanged and trims settle episodes (videos cut, npz sliced — points are stored in the per-frame view1 camera frame so no re-projection is needed — and view1/view2 T_cam_world analytically corrected to the rest pose; validated to 0.000 mm vs simulator ground truth). 97.1% of episodes remain usable.

Subtask split (robocasa)

robocasa/subtask_split/ is a per-subtask re-segmentation of the robocasa point-track episodes: each multi-step episode is split into its constituent subtasks, so one episode becomes a sequence of subtask clips — each with its own instruction and its own temporally-sliced 3D point tracks, video, and calibration.

• —Coverage: 24,504 robocasa *Pretrain episodes that carry subtask annotations (subtask counts 2–32, mode 4). The annotations (per-subtask instruction + time range) come from `happyhappy-jun/episode_subtask_annotations_x2_parallel` (MM:SS boundaries at fps 20). Atomic tasks and episodes without annotations are not included.
• —Shards: 21 tars subtask_split-000.tar … subtask_split-020.tar (~620 GB total) — same reassembly/extract as the other splits. manifest.json (episode → shard, per-subtask frame ranges + instructions) lists everything without untarring.
• —Layout — each episode dir holds one subdir per subtask:

  <episode_id>/
    subtask_00/   view1.mp4 view2.mp4 ego.mp4  robot.npz gripper.npz object*.npz  meta.json
    subtask_01/   ...
    ...

Every subtask_XX/ is a complete, self-contained sample in the exact same format as a normal episode (3D points in the view1 camera frame, identical meta.json calibration fields), sliced to that subtask's frames. Its meta.json → lang is the subtask instruction and n_frames is the subtask length.

• —Lossless tiling: the subtasks tile the original episode's [0, n_frames) contiguously — no gaps, no overlap. Concatenating an episode's subtask_00 … subtask_k in order exactly reconstructs the original: the .npz point arrays and per-frame extrinsics are byte-identical (np.array_equal) to the un-split episode, and Σ subtask n_frames == original n_frames. Videos are re-encoded per subtask with libx264 crf 18 (visually lossless — the point tracks + calibration are the exact, bit-preserved data).

Rebuild an original episode's tracks:

  import numpy as np, glob
  sds = sorted(glob.glob(f"{ep}/subtask_*"))
  robot = np.concatenate([np.load(f"{s}/robot.npz")["points"] for s in sds])  # == original robot.npz

Subtask split (droid)

droid/subtask_split/ is a per-subtask temporal split of the droid point-track episodes: each DROID episode is cut along time into its constituent subtasks, so one episode becomes an ordered sequence of subtask clips — each with its own instruction, primitive-skill label, and its own temporally-sliced 3D point tracks, video, and calibration.

• —Coverage: 28,308 episodes → 89,312 subtasks (avg 3.16 subtasks/episode; per-episode subtask counts 1–39, mode 3), spanning 15 primitive skills (pick, transfer, place, push, press, pull, slide, fold, manipulate, pour, throw, twist, insert, shake, strike).
• —Shards: 12 tars subtask_split-000.tar … subtask_split-011.tar (~327 GB total; eleven ~29.5 GB + a ~2.4 GB tail) — every shard is under HF's 50 GB limit, so each extracts directly (tar -xf subtask_split-000.tar, no .part reassembly). manifest.json (episode → shard, per-subtask frame ranges + instructions + primitive skill + RoboInter provenance) lists everything without untarring. A README.md in the same folder documents the split.
• —Layout — each episode dir holds one subdir per subtask (only the subtask_XX/ folders are shipped, not the un-split episode files):

  <episode_id>/
    subtask_00/   view1.mp4 view2.mp4  robot.npz gripper.npz  meta.json
    subtask_01/   ...
    ...

Every subtask_XX/ is a complete, self-contained sample in the exact same format as a normal droid episode (3D points in the view1 camera frame, identical meta.json calibration fields — static rig, single T_cam_world per view), sliced to that subtask's frames. Its meta.json → lang is the subtask instruction and n_frames is the subtask length; a meta.json → subtask block adds subtask_idx, frame_range, primitive_skill, instruction_add (the whole-episode instruction), and RoboInter provenance.

• —Lossless tiling: the subtasks tile the original episode's [0, n_frames) contiguously — no gaps, no overlap. Concatenating an episode's subtask_00 … subtask_k in order exactly reconstructs the original: the .npz point arrays are byte-identical (np.array_equal) to the un-split episode, and Σ subtask n_frames == original n_frames. Videos are re-encoded per subtask with libx264 crf 18 (visually lossless — the point tracks + calibration are the exact, bit-preserved data).

  import numpy as np, glob
  sds = sorted(glob.glob(f"{ep}/subtask_*"))
  robot = np.concatenate([np.load(f"{s}/robot.npz")["points"] for s in sds])  # == original robot.npz

• —Annotation source: the per-subtask segmentation (time boundaries, subtask instruction, primitive-skill label) comes from `InternRobotics/RoboInter-Data` (InternRobotics, MIT license), Annotation_pure — please cite it. Annotations are joined 1:1 to DROID episodes on the DROID lab + timestamp (the episode id).

Visualizer

tools/visualize_episode.py renders a row-concatenated multi-view video with the 3D point tracks re-projected onto every view using only the stored calibration — use it to verify calibration/npz consistency:

python tools/visualize_episode.py --split droid --episode 0 --root <extracted_root>
python tools/visualize_episode.py --split hrdexdb --episode allegro_v5__apple__3 --root <extracted_root>

--episode accepts an integer index or an episode id. Dependencies: numpy, opencv-python. Example outputs are in tools/examples/.

Verification pairs

droid/debug_lang_pairs/ contains side-by-side fail-vs-success videos demonstrating the failure-instruction borrowing (failure left/red, same-session success donor right/green) with pair_info.json provenance for each.

Provenance & licenses

• —droid: derived from DROID (CC-BY-4.0); language annotations merged from the official aggregated annotations and the KarlP/droid 75k release.
• —robocasa: derived from RoboCasa datasets.
• —hrdexdb: HRDexDB multi-camera teleoperation recordings.
• —Success labels come from the source datasets' own labels/path conventions; 8 label-contradictory droid episodes were excluded entirely.