benudavis/cluttered_grasp

Cluttered Grasp (UR10 + Robotiq 2F-140 + tactile + depth)

Real-robot teleop demonstrations of cluttered grasping. Episodes are gated on gripper_tcp height in base_link (record while low, pause while high), time-synced to the RealSense RGB master @ 30 Hz, then converted to LeRobot format.

Sensors & resolutions

• —Master clock: compressed RGB frames — one dataset row per RGB stamp (~30 Hz).
• —Sync: nearest-neighbor within ~50 ms, then hold-last for depth/tactile/proprio.
• —Depth (logged & exported): native D4xx depth at 848×480 from depth/image_rect_raw into bags → zarr center_cam_depth → LeRobot. Unit millimetres (uint16). Not the 1280×720 aligned_depth_to_color stream (that is used at runtime for ContactGraspNet only).
• —Depth vs RGB: depth rows are time-synced to RGB but not pixel-aligned — fuse with extrinsics / reprojection, not naive (u,v) overlay on center_cam.
• —Depth export for this dataset: 12-bit `gray12le` HEVC (video12), log-quantized over 0.2–1.0 m.

Dataset structure

cluttered_grasping/
├── meta/
│   ├── info.json          # schema, fps, totals, video codec info
│   ├── stats.json         # per-feature normalization stats
│   ├── tasks.parquet      # task prompts
│   ├── episodes/          # per-episode metadata
│   └── tactile_refs/      # pre-trim tactile Δ baselines (one .npz per episode)
│       ├── 000000.npz
│       ├── 000001.npz
│       └── …              # {episode_index:06d}.npz
├── data/
│   └── chunk-*/file-*.parquet   # state, action, indices
└── videos/
    ├── observation.images.center_cam/chunk-*/file-*.mp4        # AV1 1280×720 (W×H)
    ├── observation.images.center_cam_depth/chunk-*/file-*.mp4  # HEVC gray12le unaligned depth 848×480 (W×H)
    ├── observation.images.tactile_L/chunk-*/file-*.mp4         # AV1, 320×240
    └── observation.images.tactile_R/chunk-*/file-*.mp4

Tactile reference sidecars (meta/tactile_refs/)

Training subtracts a per-episode tactile baseline before letterboxing (Δ tactile). LeRobot video rows are leading-trimmed at export (home hold / NaN policy dropped), but the ref is computed from the raw source zarr at frame 0 so it matches labeling, deploy, and live recording (mean of the first 10 non-empty tactile frames at bag start).

One compressed NumPy archive per LeRobot episode_index:

Each .npz file:

Not a LeRobot parquet/video feature — load sidecars by episode_index when preprocessing tactile (e.g. cluttered-grasp-planner training/tactile_refs.py). Written on full convert and via zarr_to_lerobot.py --refs-only (no video re-encode). Include `meta/tactile_refs/` on HF upload if you train with sidecar-based Δ tactile.

Features (per frame)

observation.state EE pose is gripper_tcp in base_link. action pose delta: delta_xyz = pos[t+1]-pos[t]; delta_rpy = euler_xyz(R[t+1] @ R[t].T) (world/fixed-frame). Gripper on both action channels is discrete mode (-1 open, 0 hold, +1 close), not absolute position. Under the hood teleop/CGN ramp a setpoint (logged in zarr as gripper_setpoint) with the same controller. Wrist force/torque is not exported to LeRobot (still present in upstream zarr if needed).

Label example: terminal phases … success, success → next.done = 1 on the last exported frame; next.success = 1 on that frame too. Failed episodes still get next.done = 1 on the last frame; next.success = 0.

Episode definition

• —Start: cgn_teleop_trial accepts a grasp preview and calls begin_episode (bag warm-up at home, then approach).
• —End: EE reaches lift handoff (LIFT_Z for teleop, or after auto lift); end_episode then place/home are not recorded.
• —No recording between trials.
• —Fully automated vs human: check intervention max / zarr attr intervention_used.

Episode composition

Source zarrs are converted in sorted path order, so later collection days sit at the end of the LeRobot index range.

The last 90 + 194 = 284 episodes are recovery-only. Slice with episode_index >= 710 (or 710:800 / 800:994 for the two recovery days). Earlier episodes are not recovery-tagged; they mix all trial types.

Load

from lerobot.datasets.lerobot_dataset import LeRobotDataset

ds = LeRobotDataset("benudavis/cluttered_grasp")
print(ds)
frame = ds[0]
# EE state / action:
#   frame["observation.state"]         # (8,) eef xyz+quat + gripper
#   frame["action"]                   # (7,) delta_xyz + delta_rpy + abs gripper next
#   frame["observation.joint_state"]  # (7,) joints + gripper
#   frame["observation.joint_action"] # (7,) next joints + gripper
#   frame["observation.cgn_grasp_pose"]  # (7,) constant CGN target
# RGB / tactile: frame["observation.images.center_cam"], ...
# Depth: frame["observation.images.center_cam_depth"]  # 480×848×1 mm, unaligned 848×480 capture
# Phases / returns:
#   frame["episode_state"]    # 0 reach / 1 grasp_lift
#   frame["intervention"]     # 1 if teleop used
#   frame["grasp_state"]      # int64 phase id (contact labels)
#   frame["grasp_sequence"]   # episode class: 0 other, 1 scl, 2 ucl, 3 ucl→scl, …
#   frame["next.success"]     # 1.0 on last exported frame if episode ends in success
#   frame["next.done"]        # 1.0 on last exported frame (every episode)

Collection / conversion notes

• —ROS bags → one zarr episode per bag (bag2dataset under ~/venvs/clutter, zarr≥3).
• —Episodes are service-gated by cgn_teleop_trial (begin after preview Enter with warm-up; end at lift handoff).
• —Offline rebuild: scripts/zarr_to_lerobot.py --fps 30 --depth-mode video12 over ~/EDG_Experiment/clutter/**/*.zarr (skips clutter/old/).
• —Phase labels: scripts/label_grasp_states.py (Streamlit) writes data/obs/policy/grasp_state and, on Save / Save all, derived next_success / next_done on each episode zarr.
• —Converter sets next.done / next.success with the same terminal-frame rules as the labeler (last frame; success from terminal grasp_state). grasp_sequence is derived from the trimmed closure chain (see package README). Existing datasets: zarr_to_lerobot.py --labels-only patches labels + adds grasp_sequence without re-encoding videos. zarr_to_lerobot.py --refs-only writes or refreshes meta/tactile_refs/*.npz only.
• —Trial stats: python scripts/plot_trial_outcomes.py --zarr-root … --out … (or zarr_to_lerobot.py --trial-stats-out …).
• —Bag convert also skips leading frames with empty tactile / NaN policy before writing zarr.
• —Auto-label may set contact/success from tactile blob area; only terminal success runs (touching episode end) stay success.
• —Actions: EE delta pose + absolute next gripper; joint absolute next (see scripts/lerobot_convert/actions.py).
• —Upload: scripts/upload_lerobot_hf.py (copies this card to the dataset root as README.md).