yuezh23/UMI-Gripper-Grasping

UMI-Gripper-Grasping

This repository contains the datasets used for the project UMI-Gripper-Grasping, which investigates contact-aware grasp regulation for deformable object manipulation using GelSight tactile sensing, force/torque sensing, and leader–follower teleoperation.

The repository contains two sub-datasets:

• —Force-State Training Data – labeled tactile and force recordings used to train the three-class XGBoost force-state classifier.
• —LeRobot-UMI-Integration – teleoperation and closed-loop grasping experiments collected using the integrated LeRobot and UMI sensing system.

Force-State Training Data

This directory contains the labeled UMI-gripper sensor recordings used to train the three-class force-state classifier.

Dataset Organization

• —7 Objects: bottle_cap, chips_can, foam_brick, marker, orange, paper_cup, and softball
• —3 Labels:
• —low → grasping force is too low (too_low)
• —medium → appropriate grasping force (fine)
• —high → grasping force is too high (too_high)
• —Each object-label combination contains 20 independent trials (01–20), resulting in 420 recorded episodes.

Directory Structure

train/<object>/<label>/<trial>/
└── episodes/<episode_id>/
    ├── episode_info.json
    ├── metadata.json
    ├── recording_force_mms101_left.jsonl
    ├── recording_force_mms101_right.jsonl
    ├── recording_gelsight_left.jsonl
    ├── recording_gelsight_right.jsonl
    ├── synced_data.jsonl
    └── images/
        ├── gelsight_left/
        └── gelsight_right/

Recorded Data

Each MMS101 record contains

• —Force: fx, fy, fz
• —Torque: tx, ty, tz

synced_data.jsonl aligns the left/right force-torque measurements with the nearest left/right GelSight frames to generate synchronized sensor records for feature extraction and XGBoost model training.

LeRobot-UMI-Integration

LeRobot-UMI-Integration/ contains experiments collected using the integrated LeRobot + UMI system. Each object folder contains multiple independent experimental trials.

Experiment Types

1. Manual Teleoperation

manual_teleoperate/<object>/<trial>/

Leader–follower teleoperation used as the human-operation baseline.

The operator directly controls the follower gripper opening through the leader gripper.

Each trial contains

• —LeRobot observations and commanded actions
• —Controller logs
• —High-view video
• —Low-view video
• —Right-wrist video

2. Closed-loop Grasp-and-Hold (Main Experiments)

right_gripper_hold/<object>/<trial>/

The gripper starts from a loose state and automatically adjusts its aperture using the XGBoost force-state classifier until the predicted state becomes fine. The robot then lifts or holds the object.

Objects include

• —bottle cap
• —chips can
• —foam brick
• —paper box
• —paper cup
• —plastic bottle
• —soft ball 1
• —soft ball 2
• —tape
• —tissue box

3. Closed-loop Release

right_gripper_loose/<object>/<trial>/

The object is initially grasped with excessive force. The controller incrementally releases or re-tightens the gripper until the predicted force state returns to fine.

Objects include

• —chips can
• —plastic bottle
• —soft ball 2

4. Gripper Scanning Experiments

right_gripper_scan/soft_ball2/

Two fixed-rate scanning experiments are provided.

loosetotight

Five trials gradually close the gripper while recording the state transition

too_low → fine → too_high

tighttoloose

Five trials gradually open the gripper until the object drops while recording the state transition

too_high → fine → too_low

Trial Directory Structure

<experiment>/<object>/<trial>/
├── controller.log
├── lerobot/
│   ├── data/
│   ├── meta/
│   └── videos/
└── sensors/
    └── episodes/

lerobot/

Contains robot-side recordings:

• —data/ – robot observations and commanded actions (Parquet)
• —meta/ – dataset metadata, run configuration, and controller/model records
• —videos/ – high-view, low-view, left-wrist, and right-wrist MP4 videos

sensors/

Contains UMI sensor recordings:

• —GelSight image sequences
• —GelSight timestamps
• —MMS101 force/torque measurements
• —Synchronized sensor records