flex-pi/self_repair_gripper_dagger

selfrepairgripper_dagger

Robot self-repair, DAgger rollouts with operator corrections on the same task as self_repair_gripper_bc.

Real-robot bimanual manipulation data collected on a YAM arm pair, released as part of the Flex-π project. Stored in LeRobot v2.1 format with synchronized RGB and metric depth from three cameras.

At a glance

Tasks

1. 1.Pick up the gripper from the table and insert it into the empty holder that is missing its gripper. Then pick up a screw, insert it into the mounting hole, and use the screwdriver to tighten it. Finally, clean up the table by picking up the vegetable and placing it into the bucket.

Layout

meta/
  info.json              # feature schema, totals, chunking
  tasks.jsonl            # task_index -> natural-language instruction
  episodes.jsonl         # per-episode length + task
  episodes_stats.jsonl   # per-episode min/max/mean/std for state & action
  camera_intrinsics.json # per-camera pinhole K at stored resolution
data/chunk-{NNN}/episode_{NNNNNN}.parquet
videos/chunk-{NNN}/observation.images.{cam}/episode_{NNNNNN}.mp4   # RGB
videos/chunk-{NNN}/observation.depth_ffv1.{cam}/episode_{NNNNNN}.mkv # depth

Episodes are indexed 0 .. 2153, chunked at 1000 episodes (3 chunks). The parquet index column is a global frame counter running 0 .. 609,384 across the whole dataset.

Camera intrinsics

Pinhole K at the stored 640×360 resolution, averaged over episodes:

Reading the depth

The depth streams are an extension to stock LeRobot. They are declared with dtype: "depth_video" (not "video") in meta/info.json precisely so that the stock LeRobotDataset loader skips them — you get a working RGB dataset out of the box, and depth needs the decoder below.

Each depth frame is a single-channel uint16, millimetre map, FFV1-encoded in gray16le inside a Matroska container. 0 means no return. To decode a frame:

import av, numpy as np

with av.open("videos/chunk-000/observation.depth_ffv1.cam_high/episode_000000.mkv") as c:
    for frame in c.decode(video=0):
        depth_mm = frame.to_ndarray(format="gray16le").astype(np.uint16)  # (H, W)
        depth_m = depth_mm.astype(np.float32) / 1000.0

FFV1 is lossless, so the decoded values are bit-exact with what the sensor reported. Do not transcode these to a lossy codec.

State and action layout

observation.state and action are both 32-D. The vector is grouped by field, not by arm:

The authoritative per-dimension names are in meta/info.json under features.observation.state.names. The 6-D rotation is the first two rows of the 3×3 rotation matrix, row-major flattened (Zhou et al., On the Continuity of Rotation Representations); recover R by Gram–Schmidt on those two rows and their cross product.

Loading

RGB only, with stock LeRobot:

from lerobot.common.datasets.lerobot_dataset import LeRobotDataset
ds = LeRobotDataset("flex-pi/self_repair_gripper_dagger")

RGB + depth: use the depth-aware loader from the Flex-π codebase.

Provenance

Built from source recordings self_repair_gripper_v2.1_dagger_maniflow_r1, self_repair_gripper_dagger_maniflow_r2.

Citation

If you use this dataset, please cite the Flex-π project.