Ronaldo-GOAT/pose6daug

pose6daug

Real-world Franka manipulation episodes with object-swap and action augmentation artifacts. 120 training episodes over 4 objects (bluecup, greenpear, kanu, white_spray), dual ZED cameras (exo static + ego wrist-mounted).

Layout

Per-frame PNGs are packed into uncompressed tars per episode — the dataset has ~427k mask/plate frames and loose files hit Hugging Face's per-repo file recommendation and API rate limits hard.

data/<object>/<NNNN>/
  masks.tar               masks/{object,object_inpaint,robot,robot_sam2}/{exo,ego}/
                          plus preserved generations (object_inpaint_v1,
                          _v3dil, _v3raw, _pre_maskfix)
  inpainted_frames.tar    inpainted/{object,robot}/{exo,ego}/ plus preserved
                          plate generations (object_v1, object_pre_v3,
                          object_pre_maskfix)
  inpainted/*.mp4         per-generation encodes (loose)
  recorded_videos/        raw mp4: exo 37149196_*, ego 11845904_*
  teleoperation.h5        robot states / actions
scripts/                  pipeline code
calib_ws/                 calibration_final.json, CALIBRATION_NOTES.md
external/swap_meshes/     metric swap meshes
external/sim_action_aug_code/   MuJoCo scene + rollout code
external/twin/            digital-twin geometry
actaug/swap_videos/       40 object-swap mp4s (5 episodes)
actaug/rollout_videos/    122 rollout mp4s (<ep>__<stage>__<config>__{exo,wrist})

Unpack with tar -xf masks.tar inside the episode directory.

Raw ZED SVO stereo recordings (56.5 GB) are not in this repo yet.

Calibration

Exo intrinsics fx 523.947, cx 667.921, cy 371.911 at 1280x720. T_base_exocam t = [0.2565, 0.6050, 0.6329]; T_ee_egocam t = [-0.0839, 0.0296, 0.0261]; time_offset_s 0.27112.

Known issues

Ego object masks are unreliable on a large fraction of episodes. An audit with independent per-object detectors flagged bad frames in 27/30 bluecup, 25/30 kanu (23/30 by visual review) and 27/30 whitespray episodes, and found that 22 of 60 delivered blue_cup + white_spray ego plates still contain the original object.

Cause: both mask-seeding routes seed the SAM2 ego track at frame 0 via a geometric transfer from the exo view. When the object is not visible in the ego view at frame 0 (it sits behind the wooden riser early in many episodes) the seed lands on a gripper finger and SAM2 tracks that instead. The acceptance gate passed on the worst cases.

greenpear was rebuilt with an independent image-domain detector (1239 bad frames across 22/30 episodes -> 47 frames across 15/30). bluecup, kanu and white_spray are not yet rebuilt. Exo masks and exo plates are unaffected.

Also: the green_pear stem/leaf is excluded from the object mask on ~31% of frames (a background-negative sampling bug places a SAM2 negative prompt on the stalk).

See scripts/docs/realvideo/PIPELINE.md.