Datasetpublic

dgrachev/robotwin_extrinsics

This dataset was created using LeRobot. Dataset Description RoboTwin 2.0 (aloha-agilex) with end-effector poses and extrinsics Generated from lerobot/robotwin_unified at commit 1287871839fae2296bc27b88a5457c3e1eba8e1f by benchmarks/robotwin/augment_robotwin_dataset.py. Original state and action columns are unchanged: 14 joint drive targets [left arm(6), left gripper, right arm(6), right gripper], grippers in [0, 1] with 1 = open, at 30 Hz; the action is the… See the full description on the dataset page: https://huggingface.co/datasets/dgrachev/robotwin_extrinsics.

sourceHugging Faceapache-2.0updated 19d agoView on Hugging Face

0likes354downloads

Dataset Card

This dataset was created using LeRobot.

Dataset Description

RoboTwin 2.0 (aloha-agilex) with end-effector poses and extrinsics

Generated from lerobot/robotwin_unified at commit 1287871839fae2296bc27b88a5457c3e1eba8e1f by benchmarks/robotwin/augment_robotwin_dataset.py. Original state and action columns are unchanged: 14 joint drive targets [left arm(6), left gripper, right arm(6), right gripper], grippers in [0, 1] with 1 = open, at 30 Hz; the action is the next frame's target. The three video columns are renamed observation.images.static1 (head), .wrist1 (left wrist) and .wrist2 (right wrist).

The layout is 50 tasks x 550 episodes in RoboTwin's alphabetical task order; the first 50 episodes of each block are the democlean demonstrations the leaderboard trains on, the remaining 500 are demorandomized. Verified against the raw recordings of beat_block_hammer: every raw demo_clean episode is one LeRobot episode of the block, state[t] its joint target t and action[t] its target t + 1.

Pose columns hold [position, rotation, gripper] per arm, left then right. Rotations are active, right-handed R_world_from_end_effector of the arm's link6, which is RoboTwin's endpose (get_obs()["endpose"], the pose take_action(action_type="ee") plans to), by forward kinematics over the URDF from the stored joint drive targets. Against the recorded physical links that is exact when an arm rests and off by the controller's tracking gap while it moves: millimetres in free motion, 27 mm / 2.8 deg at worst on the validation task, at the hammer strike. rot6d stores the first two columns; rot6d_rows stores the first two rows.

—observation.state.ee_<rotation>: the pose at the frame's drive targets with the gripper state, 2 x (4 + rotation width) wide.
—action.ee_absolute.<rotation>: the pose the action commands next with the gripper command.
—action.ee_delta.<rotation>: the same commands as per-step displacements from the state pose, in world axes and metric units (p_cmd - p_state, R_cmd @ R_state.T) with the gripper command.
—action.joint and observation.state.joint: the original columns with flat names, so relative_exclude_joints can find the grippers. The action.joint statistics describe 50-step chunks minus the state at chunk start (grippers absolute), the values pi05 normalizes under use_relative_actions; the original action keeps absolute statistics.
—observation.intrinsics.<camera>: 3x3 matrices for the stored 640x480 frames, from the D435's 37 degree vertical field of view (the recordings are 320x240 renders of the same cameras).
—observation.extrinsics.<camera>: world-from-camera [xyz, quaternion xyzw] in the OpenCV convention at every timestamp; the head camera from the embodiment config (exactly the recorded pose), the wrist cameras by forward kinematics of their URDF links.
—observation.gripper.<arm>_time_to_close and _time_to_open: seconds until the next command transition of that arm in the same episode; NaN means no remaining event.

meta/stats.json holds histogram quantiles over every frame for every numeric column (the source had none for action and observation.state). It additionally carries per-offset statistics for chunk-anchored actions under action.ee_absolute.<rotation>.anchored_horizon_50, frame-expressed ones under .expressed_<frame>_horizon_50 for base, static1, wrist1, wrist2 (one frame for both arms) and wrist (each arm in its own wrist camera) plus the pooled .expressed_horizon_50, and anchored-then-expressed ones pooled over the same frames under .anchored_expressed_horizon_50. These are stats keys, not dataset columns, so normalization ignores them unless a config asks for them by name. Every statistic is shaped (50, action_dim); row i describes offset i inside the chunk, so it broadcasts directly onto a (batch, chunk_size, action_dim) action tensor and a shorter chunk slices rows :chunk_size. Every 10-th chunk start whose whole horizon stays inside one episode contributes; count is that number of chunks, times frames when pooled.

Anchoring: positions are relative to the chunk's first state position in world axes, rotations are R_action @ R_anchor.T, and the gripper stays the absolute command. Expression in a frame frozen at chunk start: absolute poses become R.T (p - t) and R.T M, displacements the pure basis change R.T v and R.T M R.

Published to main and tagged v3.0, which is the revision LeRobot resolves when a config names no revision. Pin a resolved commit SHA in train and evaluation configs to keep a run reproducible against later updates.

—Homepage: [More Information Needed]
—Paper: [More Information Needed]
—License: apache-2.0

Dataset Structure

meta/info.json:

json

{
    "codebase_version": "v3.0",
    "fps": 30,
    "features": {
        "observation.state": {
            "dtype": "float32",
            "shape": [
                14
            ],
            "names": {
                "motors": [
                    "left_waist",
                    "left_shoulder",
                    "left_elbow",
                    "left_forearm_roll",
                    "left_wrist_angle",
                    "left_wrist_rotate",
                    "left_gripper",
                    "right_waist",
                    "right_shoulder",
                    "right_elbow",
                    "right_forearm_roll",
                    "right_wrist_angle",
                    "right_wrist_rotate",
                    "right_gripper"
                ]
            },
            "fps": 30
        },
        "action": {
            "dtype": "float32",
            "shape": [
                14
            ],
            "names": {
                "motors": [
                    "left_waist",
                    "left_shoulder",
                    "left_elbow",
                    "left_forearm_roll",
                    "left_wrist_angle",
                    "left_wrist_rotate",
                    "left_gripper",
                    "right_waist",
                    "right_shoulder",
                    "right_elbow",
                    "right_forearm_roll",
                    "right_wrist_angle",
                    "right_wrist_rotate",
                    "right_gripper"
                ]
            },
            "fps": 30
        },
        "timestamp": {
            "dtype": "float32",
            "shape": [
                1
            ],
            "names": null
        },
        "frame_index": {
            "dtype": "int64",
            "shape": [
                1
            ],
            "names": null
        },
        "episode_index": {
            "dtype": "int64",
            "shape": [
                1
            ],
            "names": null
        },
        "index": {
            "dtype": "int64",
            "shape": [
                1
            ],
            "names": null
        },
        "task_index": {
            "dtype": "int64",
            "shape": [
                1
            ],
            "names": null
        },
        "observation.state.ee_rotvec": {
            "dtype": "float32",
            "shape": [
                14
            ],
            "names": null
        },
        "observation.state.ee_quat_xyzw": {
            "dtype": "float32",
            "shape": [
                16
            ],
            "names": null
        },
        "observation.state.ee_rot6d": {
            "dtype": "float32",
            "shape": [
                20
            ],
            "names": null
        },
        "observation.state.ee_rot6d_rows": {
            "dtype": "float32",
            "shape": [
                20
            ],
            "names": null
        },
        "observation.state.ee_matrix": {
            "dtype": "float32",
            "shape": [
                26
            ],
            "names": null
        },
        "observation.state.ee_euler_xyz_fixed": {
            "dtype": "float32",
            "shape": [
                14
            ],
            "names": null
        },
        "action.ee_absolute.rotvec": {
            "dtype": "float32",
            "shape": [
                14
            ],
            "names": null
        },
        "action.ee_absolute.quat_xyzw": {
            "dtype": "float32",
            "shape": [
                16
            ],
            "names": null
        },
        "action.ee_absolute.rot6d": {
            "dtype": "float32",
            "shape": [
                20
            ],
            "names": null
        },
        "action.ee_absolute.rot6d_rows": {
            "dtype": "float32",
            "shape": [
                20
            ],
            "names": null
        },
        "action.ee_absolute.matrix": {
            "dtype": "float32",
            "shape": [
                26
            ],
            "names": null
        },
        "action.ee_absolute.euler_xyz_fixed": {
            "dtype": "float32",
            "shape": [
                14
            ],
            "names": null
        },
        "action.ee_delta.rotvec": {
            "dtype": "float32",
            "shape": [
                14
            ],
            "names": null
        },
        "action.ee_delta.quat_xyzw": {
            "dtype": "float32",
            "shape": [
                16
            ],
            "names": null
        },
        "action.ee_delta.rot6d": {
            "dtype": "float32",
            "shape": [
                20
            ],
            "names": null
        },
        "action.ee_delta.rot6d_rows": {
            "dtype": "float32",
            "shape": [
                20
            ],
            "names": null
        },
        "action.ee_delta.matrix": {
            "dtype": "float32",
            "shape": [
                26
            ],
            "names": null
        },
        "action.ee_delta.euler_xyz_fixed": {
            "dtype": "float32",
            "shape": [
                14
            ],
            "names": null
        },
        "observation.gripper.left_time_to_close": {
            "dtype": "float32",
            "shape": [
                1
            ],
            "names": null
        },
        "observation.gripper.left_time_to_open": {
            "dtype": "float32",
            "shape": [
                1
            ],
            "names": null
        },
        "observation.gripper.right_time_to_close": {
            "dtype": "float32",
            "shape": [
                1
            ],
            "names": null
        },
        "observation.gripper.right_time_to_open": {
            "dtype": "float32",
            "shape": [
                1
            ],
            "names": null
        },
        "observation.intrinsics.static1": {
            "dtype": "float32",
            "shape": [
                3,
                3
            ],
            "names": null
        },
        "observation.extrinsics.static1": {
            "dtype": "float32",
            "shape": [
                7
            ],
            "names": null
        },
        "observation.intrinsics.wrist1": {
            "dtype": "float32",
            "shape": [
                3,
                3
            ],
            "names": null
        },
        "observation.extrinsics.wrist1": {
            "dtype": "float32",
            "shape": [
                7
            ],
            "names": null
        },
        "observation.intrinsics.wrist2": {
            "dtype": "float32",
            "shape": [
                3,
                3
            ],
            "names": null
        },
        "observation.extrinsics.wrist2": {
            "dtype": "float32",
            "shape": [
                7
            ],
            "names": null
        },
        "action.joint": {
            "dtype": "float32",
            "shape": [
                14
            ],
            "names": [
                "left_waist",
                "left_shoulder",
                "left_elbow",
                "left_forearm_roll",
                "left_wrist_angle",
                "left_wrist_rotate",
                "left_gripper",
                "right_waist",
                "right_shoulder",
                "right_elbow",
                "right_forearm_roll",
                "right_wrist_angle",
                "right_wrist_rotate",
                "right_gripper"
            ]
        },
        "observation.state.joint": {
            "dtype": "float32",
            "shape": [
                14
            ],
            "names": [
                "left_waist",
                "left_shoulder",
                "left_elbow",
                "left_forearm_roll",
                "left_wrist_angle",
                "left_wrist_rotate",
                "left_gripper",
                "right_waist",
                "right_shoulder",
                "right_elbow",
                "right_forearm_roll",
                "right_wrist_angle",
                "right_wrist_rotate",
                "right_gripper"
            ]
        },
        "observation.images.static1": {
            "dtype": "video",
            "shape": [
                480,
                640,
                3
            ],
            "names": [
                "height",
                "width",
                "rgb"
            ],
            "info": {
                "video.fps": 30.0,
                "video.height": 480,
                "video.width": 640,
                "video.channels": 3,
                "video.codec": "av1",
                "video.pix_fmt": "yuv420p",
                "video.is_depth_map": false,
                "has_audio": false
            }
        },
        "observation.images.wrist1": {
            "dtype": "video",
            "shape": [
                480,
                640,
                3
            ],
            "names": [
                "height",
                "width",
                "rgb"
            ],
            "info": {
                "video.fps": 30.0,
                "video.height": 480,
                "video.width": 640,
                "video.channels": 3,
                "video.codec": "av1",
                "video.pix_fmt": "yuv420p",
                "video.is_depth_map": false,
                "has_audio": false
            }
        },
        "observation.images.wrist2": {
            "dtype": "video",
            "shape": [
                480,
                640,
                3
            ],
            "names": [
                "height",
                "width",
                "rgb"
            ],
            "info": {
                "video.fps": 30.0,
                "video.height": 480,
                "video.width": 640,
                "video.channels": 3,
                "video.codec": "av1",
                "video.pix_fmt": "yuv420p",
                "video.is_depth_map": false,
                "has_audio": false
            }
        }
    },
    "total_episodes": 27500,
    "total_frames": 6075103,
    "total_tasks": 23559,
    "chunks_size": 1000,
    "data_files_size_in_mb": 100,
    "video_files_size_in_mb": 200,
    "data_path": "data/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet",
    "video_path": "videos/{video_key}/chunk-{chunk_index:03d}/file-{file_index:03d}.mp4",
    "robot_type": "aloha",
    "splits": {
        "train": "0:27500"
    }
}

Citation

BibTeX:

bibtex

[More Information Needed]