Orboh/umi-okra-dex1

UMI Okra Grasping Dataset — Lab Subset (Unitree Dex1-1)

Universal Manipulation Interface (UMI) hand-held teleoperation data for an okra-fruit grasping / harvesting task. Recorded to train a Diffusion Policy (with a comparison ACT track) deployed on a Unitree G1.

This is the indoor lab subset. Every session here was recorded in a lab mock okra field (artificial foliage, white-walled room). The outdoor sessions from the original collection are not included — see Scope and sanitisation below.

⚠️ LICENSE PENDING. Redistribution terms have not been settled yet. Do not treat this copy as released until this block is replaced with an actual license.

Scope and sanitisation

This release is a filtered, sanitised derivative of the internal collection. What was done:

People in frame — open item

The lab is a working office. A person-detection pass over all 1,262 clips (2 fps, 1280 px, conf 0.15) flagged 123 clips, and the highest-confidence ones show colleagues standing, walking and sitting at desks at close range — identifiable at the native 2704×2028. Those 123 clips are not in this tree; they are held out pending a decision on each one. GoPro's own FACE telemetry was not used: it returned 0 detections on a clip that plainly shows people, and fired on foliage in an empty room.

Currently withheld, by session:

The packed dataset.zarr.zip files were screened separately, frame by frame (97,784 frames). Detections there resolve to the operator's own hand on the gripper and to static background objects (chair and monitor edges); at 224×224 nothing in them is identifiable. The five zips ship unmodified.

What was not changed: the video streams themselves are remuxed, never re-encoded, so pixels are bit-identical to the source. IMU (ACCL / GYRO / CORI / IORI / GRAV) is untouched — only GPS5 was deleted, and the UMI/ORB-SLAM3 pipeline never reads it.

Consequence for re-processing: because the GPMF track is gone, UMI's scripts_slam_pipeline/01_extract_gopro_imu.py cannot regenerate imu_data.json from these videos. The already-extracted imu_data.json ships alongside each demo, so steps 02 onward run normally.

Collection rig

• —Gripper: UMI GoPro hand-held gripper, modified to fit a Unitree Dex1-1 hand. Fingertip ArUco markers (DICT_4X4_50, 16 mm, id 0 = left / id 1 = right) plus a mirror for implicit stereo.
• —Gripper open/close is NOT actuated in this task (grip width ≈ constant), so the policy action space is 9-dim (pos3 + rot6d); gripper_width is effectively a constant column.
• —Camera: GoPro HERO9 / HERO10 with Max Lens Mod circular fisheye, 2.7K @ ~60 fps (gopro_intrinsics_2_7k.json).

Pipeline

• —Repo universal_manipulation_interface, branch learning-diffusion. Dex1-1 rig dimensions baked in: cam_to_center_height=0.061, cam_to_mount_offset=0.035, tcp_offset=0.173, nominal_z=0.093.
• —ORB-SLAM3 (Docker chicheng/orb_slam3), steps 00–07 → dataset.zarr.zip.
• —Deploy: Unitree G1 (rt/arm_sdk → click → IK reach → UMI diffusion fine-adjust).

Layout

Each session directory contains (typical):

• —dataset.zarr.zip — UMI ReplayBuffer (RGB + EEF 6DOF pose + gripper_width). The artifact consumed directly by training.
• —dataset_plan.pkl — SLAM / plan.
• —demos/demo_<serial>_<timestamp>/ — per-episode; demos/mapping/ is the SLAM mapping clip; EXCLUDED_* are dropped takes.
• —raw_videos/ (some sessions) — .LRV low-res proxies and .THM thumbnails, slam_feature_viz/ — QC plots.

Root-level *.py / *.sh are the analysis / gate-check scripts used during collection (gate_check.sh, tape_zone_quant.py, etc.). Their host-specific paths were replaced with ${UMI_REPO} / ${UMI_ENV}; set those to your own checkout and conda env.

Two things to know about this Hub copy

*`raw_videos/.MP4 became rawvideosindex.csv.** In the internal tree those entries are symlinks onto demos/…/raw_video.mp4, so uploading them listed every clip twice — once under its original GoPro name (GX010190.MP4), once under demos/`. That doubled the repo for zero extra footage. They are replaced by a single lookup table at the root:

session,gopro_name,demo_path
okra_20260717_s01,GX010041.MP4,okra_20260717_s01/demos/demo_<serial>_<ts>/raw_video.mp4

demos/ is canonical. Use the CSV when you need to go from a GoPro filename back to a clip. The .LRV proxies and .THM thumbnails that also live under raw_videos/ are distinct files, not duplicates, and they ship as-is.

Unpacked `dataset.zarr/` stores are deliberately absent. A zarr chunk directory here holds 11,411 files in data/camera0_rgb/, and the Hub hard-rejects any commit that puts more than 10,000 files in a single directory. A zarr chunk directory cannot be resharded without breaking the format, so only the packed form is published. Nothing is lost: the dataset.zarr.zip beside each holds the identical entries, and it is the artifact training consumes anyway. Unpack locally if you need the directory form.

Sessions

Sessions with a compiled dataset.zarr.zip (directly trainable):

Raw / intermediate sessions (kept for reproducibility; not all compiled to a replay buffer):

• —2026-07-31_okura300data (raw GoPro, 4.1G; only ~6.7% usable SLAM), 2026-08-03_okura80data (63.8% after slower motions).
• —2026-08-05_okura_mapping / _mapping2 / _1330_s04 — SLAM mapping-clip sessions.
• —2026-08-05_okura_tape / 2026-08-06_okura_tape3 — cloth-tape wall texturing (tape kept ON during demos; removing it dropped yield).
• —2026-08-06_okura_backwall2 / 3 / 5 — back-wall texturing iterations.
• —okra_20260722_s02 — SLAM init failed (0 maps).

Many sessions exist because of iterative work to raise SLAM re-localization yield (target ~80%) and to fix ArUco fingertag detection.

Important caveats

• —tcp_pose is in SLAM-local coordinates (±0.3 m around origin), not robot-root.
• —Pose representation is relative.
• —gripper_width is effectively constant (open/close out of scope for this task).
• —Some sessions have a degenerate gripper_range.json (sub-mm range) — verify before use.
• —Clips have no audio track and no GoPro telemetry track by design (see Scope and sanitisation). Tools that assume a GoPro-native MP4 layout may need adjusting.

Loading a session

import zarr
# after: hf download <repo_id> 2026-08-07_okura_canary/dataset.zarr.zip --repo-type dataset
root = zarr.open("dataset.zarr.zip", mode="r")
print(root.tree())