chengyuanshu98/Grasp_both

Bimanual Storage Task — Aligned Ego + UMI Dataset

Bimanual tabletop manipulation dataset with synchronized ego (head-mounted) and UMI (wrist-mounted) cameras. 50 episodes of a two-handed storage/organization task, each ~32 seconds.

Task

双手收纳 (Bimanual Storage): An operator uses two FastUMI Pro grippers to pick, move, and place objects on a tabletop. A head-mounted ego camera records a continuous third-person overhead view of the entire workspace.

Scene

Hardware

All three cameras: 1280x1280 resolution, ~200° DFOV fisheye (SEUCM model), 60 fps RGB, 500 Hz SLAM, 500 Hz IMU.

Ground Truth for KILO

This dataset is designed to provide ground truth for validating KILO's height/gauge estimation pipeline. The following quantities are independently measured or computed with high confidence and can serve as ground truth:

Measured (physical measurement)

Computed — high confidence (can be treated as ground truth)

Derived (from the above)

What KILO can test against this dataset

1. 1.z-estimator validation: The ego height above floor (135.7 cm) is computed via the strongest non-contact estimator (§6d #3). KILO's weaker estimators (mono depth, anthropometric prior) can be benchmarked against it.
2. 2.Contact-event estimator: UMI gripper-close events + merged trajectory Z give per-surface contact heights. With known table height (72.5 cm), these can be checked for absolute accuracy.
3. 3.Inter-hand gauge: The ArUco-derived inter-camera distance (49.5 cm per episode) is the ground truth for KILO's inter-hand relative pose, independent of any SLAM drift.
4. 4.Height conditioning: The per-episode ego height variation (115–158 cm, reflecting operator posture) provides natural height diversity for testing z-conditioned augmentation.
5. 5.Surface-cluster detection: Multiple grasp/release events per episode at known table height (72.5 cm) enable testing the contact-event surface clustering logic.

Dataset Structure

aligned_dataset/
├── README.md
├── dataset_meta.json
│
├── episode_01/
│   ├── alignment.json              # Timing + spatial alignment + ground truth
│   │
│   ├── ego/                        # Head-mounted camera data (sliced per episode)
│   │   ├── video.mp4               # RGB video, 1280x1280, 60 fps
│   │   ├── timestamps.csv          # Per-frame timestamps (unix + steady_clock)
│   │   ├── slam_poses.txt          # 6-DOF pose trajectory, TUM format, ~500 Hz
│   │   ├── imu.txt                 # 6-axis IMU (gyro + accel), ~500 Hz
│   │   ├── transform/
│   │   │   ├── left_hand/          # MediaPipe hand tracking, 21 joints
│   │   │   │   ├── 00_wrist.csv
│   │   │   │   ├── 01_thumb_cmc.csv
│   │   │   │   ├── ...
│   │   │   │   └── 20_pinky_tip.csv
│   │   │   └── right_hand/         # Same 21 joints
│   │   │       └── ...
│   │   ├── calibration/
│   │   │   ├── rgb_intrinsic.json  # SEUCM fisheye camera model
│   │   │   └── rgb2imu_extrinsic.json
│   │   └── metadata.json           # SDK/firmware versions
│   │
│   └── umi -> /path/to/session     # Symlink to UMI session data
│       ├── left_hand_.../
│       │   ├── RGB_Images/
│       │   │   ├── video.mp4       # Wrist RGB, 1280x1280, 60 fps
│       │   │   └── timestamps.csv
│       │   ├── SLAM_Poses/
│       │   │   ├── slam_processed.txt   # Raw SLAM, TUM format, ~500 Hz
│       │   │   └── slam_raw.txt
│       │   ├── Merged_Trajectory/
│       │   │   └── merged_trajectory.txt  # Fused trajectory (EE frame)
│       │   └── Clamp_Data/
│       │       └── clamp_data_tum.txt     # Gripper aperture, ~200 Hz
│       ├── right_hand_.../          # Same structure
│       ├── relative_transforms_left_to_right.txt
│       ├── relative_transforms_right_to_left.txt
│       └── quality_report/
│
├── episode_02/
├── ...
└── episode_50/

Data Channels

Ego (head-mounted)

UMI (wrist-mounted, per hand)

Alignment

Ego and UMI data are recorded on separate hosts. Alignment uses wall-clock timestamps:

• —Ego: record_unix_time_ms in timestamps.csv (UNIX epoch, millisecond precision)
• —UMI: collection_summary.txt start time (wall-clock, second precision) + header_stamp (steady_clock) offset

Alignment precision: ~1 second. Each ego clip includes a ±1 second buffer around the UMI session window.

The alignment.json per episode contains the full timing correspondence and a ego_steady_to_unix_offset for converting between ego's steady_clock and UNIX timestamps.

Spatial Measurements

Each episode's alignment.json includes initial-state spatial measurements:

UMI Initial State (ArUco tag method)

A DICT4X450 ArUco tag (ID=0, 144mm) is placed on the table surface. Both UMI wrist cameras observe it in the first ~20 frames. Camera-to-tag pose via SEUCM unprojection + solvePnP gives:

Note: camera-to-table height, not EE-to-table. The camera is mounted ~14 cm above the gripper tip (Robotiq 85, non-prismatic configuration).

Ego Initial State (multi-frame triangulation)

Floor height estimated via KILO §6d estimator #3: ORB feature matching across frames with known SLAM poses → midpoint triangulation → gravity-orthogonal floor plane fit.

The per-episode variation (115–158 cm) reflects real head-height changes as the operator leans forward over the table vs. stands upright between episodes.

Camera Model

All three cameras use the Extended Unified Camera Model (EUCM/SEUCM):

Projection of point (X, Y, Z):

d = sqrt(β(X² + Y²) + Z²)
u = fx · X / (α·d + (1-α)·Z) + cx
v = fy · Y / (α·d + (1-α)·Z) + cy

Unprojection of pixel (u, v) to ray:

mx = (u - cx) / fx
my = (v - cy) / fy
r² = mx² + my²
mz = (1 - β·α²·r²) / (α·√(1 - (2α-1)·β·r²) + 1 - α)
ray = normalize(mx, my, mz)

Ego intrinsics are per-device calibrated (in calibration/rgb_intrinsic.json). UMI intrinsics use example device parameters (see note in alignment.json).

Coordinate Systems

• —SLAM frame: Gravity-aligned (Z up), origin at device boot position. Roll/pitch from IMU; x, y, z, yaw are gauge freedoms.
• —UMI EE frame (merged trajectory): Origin at gripper tip. X = forward, Y = left/right, Z = up/down.
• —Ego hand tracking: MediaPipe coordinates in camera optical frame (Z = depth from camera).

Statistics

Known Issues

• —UMI intrinsics: ArUco-based measurements use example device parameters, not per-device calibration. Systematic bias estimated at a few cm.
• —Alignment precision: Limited to ~1 second by UMI's collection_summary wall-clock resolution. Sub-second alignment possible via visual cross-correlation (ArUco tag appearance timing).
• —Ego SLAM tracking gaps: Two periods (~37–43 min and ~82–87 min into the continuous ego recording) show frozen SLAM position, likely tracking loss. Episodes during those windows still have valid RGB/IMU but degraded SLAM.