MLeggiero/g1-inspire-nut-in-box-twist2

G1 + Inspire — "pick up the tweezers and put the nut in the box" (TWIST2 high-level)

Teleoperated Unitree G1 (29-DoF) + Inspire RH56DFTP hands manipulation data, converted to LeRobot v3.0 with the TWIST2 converter (deploy_real/convert_twist2_to_lerobot.py, --action_mode high_level).

This capture adds two wrist cameras on top of the onboard ego-view — so there are three synchronized RGB streams (one head + two wrist), not the single head view of `MLeggiero/g1-inspire-pipette-tip-twist2`. The observation/action vectors are the recorder's own layout (no GR00T-style space remapping): 48-dim observation.state, 49-dim action, plus `observation.effort` (per-joint torque), `observation.force` (hand motor current) and the raw 1062-dim per-hand tactile arrays.

Read Known limitations before training on this — hand proprioception is dead and the tactile arrays need a byte swap.

Summary

• —Task (1): pick up the tweezers and put the nut in the box
• —Episodes / frames: 41 / 84,552 @ 60 fps (23.5 min); every source episode was labelled successful and none were dropped
• —Sessions: 1 capture session on 2026-08-21 (20260821_1806)
• —Cameras (3): observation.images.head_rgb (1280×720), plus observation.images.left_wrist_rgb and observation.images.right_wrist_rgb (1920×1080) — all AV1 video, 60 fps
• —observation.state: 48-dim · action: 49-dim · observation.effort: 29-dim · observation.force: 12-dim · observation.tactile.{left,right}: 1062-dim each
• —Episode length: min 1284, median 1997, mean 2062, max 3941 frames

The task

The robot grasps a pair of tweezers in its right hand, uses them to pick up a nut from the table, and places it into a box. The four right fingers stay clamped on the tweezer body while the right thumb works the tweezer's spring action — the thumb (thumb_bend, thumb_rot) is effectively the only actuated hand DoF. The left hand is idle and held open throughout.

Data fields & vector index

The 29 body joints appear everywhere in the canonical TWIST2 order: left_leg 0–5, right_leg 6–11, waist 12–14, left_arm 15–21, right_arm 22–28. Per-limb: legs are hip_pitch, hip_roll, hip_yaw, knee, ankle_pitch, ankle_roll; waist is yaw, roll, pitch; arms are shoulder_pitch, shoulder_roll, shoulder_yaw, elbow, wrist_roll, wrist_pitch, wrist_yaw.

observation.state (float32[48]) — proprioception

0:34 is the recorder's state_body; the tail is state_hand_left, state_hand_right, state_neck concatenated in that order.

action (float32[49]) — high-level teleop target

0:35 is the recorder's action_body. These are the high-level teleop targets sent to the whole-body controller, not the RL policy's motor commands — the low-level 29-dim action_low_level stream is in the source recording but is not carried here (re-run the converter with --action_mode low_level for that).

action[0:29] is the commanded target for observation.state[0:29] at the next control step. The tails are not a delayed-observation pair: state[29:34] is measured IMU, action[29:35] is commanded root motion.

observation.effort (float32[29]) — per-joint torque

Estimated joint torque (tau_est) in N·m, same canonical 29-joint order as state[0:29]. This is the motor's internal estimate, not an external torque sensor — treat it as proportional to joint load rather than a calibrated value. All 29 dims are live in all 41 episodes.

observation.force (float32[12]) — hand motor current

Inspire hand motor current per digit, force_hand_left (6) then force_hand_right (6), in the same finger order as the hand state: pinky, ring, middle, index, thumb_bend, thumb_rot. A load/current proxy, not a calibrated contact force.

observation.tactile.left / .right (int32[1062]) — raw Inspire taxels

The flat 1062-value RH56DFTP touch array per hand, un-pooled, stored as int32 (the source is uint16; int32 because parquet prefers signed integers). Region layout, in order, with grid shapes:

The stored values are byte-swapped — decode before use, see limitation 3.

Coverage after decoding (taxels that ever register, over all 41 episodes): 90/1062 left, 92/1062 right. The right hand is the working hand: its little_*, ring_*, middle_* and thumb_pad/thumb_nail regions plus palm report contact in ≥98% of frames (the tweezer-body clamp), and index_nail in 100% / index_pad in 97%. The left hand is idle, so its coverage is mostly resting/self-contact.

Other parquet columns

Cameras

Three synchronized RGB streams, all AV1 video at 60 fps:

The right wrist camera is the one looking at the manipulation (tweezers → nut → box); the left wrist camera rides the idle hand. LeRobot returns each as (3, H, W) after its default image transform.

Loading

LeRobot v3.0 layout — needs lerobot >= 0.4 (built with 0.5.1).

from lerobot.datasets.lerobot_dataset import LeRobotDataset

ds = LeRobotDataset("MLeggiero/g1-inspire-nut-in-box-twist2")
f = ds[0]
f["observation.images.head_rgb"]         # (3, 720, 1280)
f["observation.images.right_wrist_rgb"]  # (3, 1080, 1920)
f["observation.images.left_wrist_rgb"]   # (3, 1080, 1920)
f["observation.state"]                   # (48,)
f["action"]                              # (49,)
f["observation.effort"]                  # (29,)  N·m
f["observation.force"]                   # (12,)  hand motor current
f["observation.tactile.right"]           # (1062,) int32, byte-swapped — see below

Decoding tactile:

import numpy as np

def decode_taxels(v):
    """Undo the recorder's byte swap. Returns 0–4095 uint16."""
    v = np.asarray(v, dtype=np.uint16)
    return ((v & 0xFF) << 8) | (v >> 8)

Layout

meta/info.json                     LeRobot v3.0 descriptor (features, fps, paths)
meta/tasks.parquet                 the single task string
meta/episodes/chunk-000/*.parquet  per-episode length, task, file/frame ranges, per-episode stats
meta/stats.json                    aggregated per-feature mean/std/min/max/q01…q99
data/chunk-000/file-000.parquet    all 41 episodes
videos/observation.images.head_rgb/chunk-000/file-0NN.mp4         AV1
videos/observation.images.left_wrist_rgb/chunk-000/file-0NN.mp4   AV1
videos/observation.images.right_wrist_rgb/chunk-000/file-0NN.mp4  AV1

v3.0 packs many episodes per file; use meta/episodes/ to map an episode to its file and frame range rather than assuming one file per episode.

Units & conventions

Body and neck angles in radians; hand values are raw Inspire RH56DFTP register units (0–1000, 1000 = open), not radians. root_height in meters, root velocities in m/s and rad/s in the robot-local frame. Effort in N·m. Hand force in raw motor-current units. Tactile in raw (byte-swapped) taxel counts.

Known limitations

Properties of the capture, not of the conversion. Stated plainly so you can decide what to train on.

1. 1.Hand proprioception is dead. state[40:46] (right hand) is constant within every one of the 41 episodes, and state[34:40] (left hand) is constant in 41/41 episodes for ring/middle, 40/41 for pinky/index and 37/41 for the thumb dims, with negligible spread. The recorder's hand-state stream was not updating during capture. The right hand is the one doing the task, so there is no usable right-hand proprioception. Mask state[34:46].
2. 2.Action hand channel is nearly constant. action[41:47] (right hand) varies in only 2 of 6 dims — thumb_bend (19,632 distinct values) and thumb_rot (2,487) — the tweezer spring action. The four fingers take 98 distinct values across the whole dataset but are constant within 38/41 episodes (index constant in 41/41): they are pinned on the tweezer body. action[35:41] (left hand) is fully constant at [1000, 1000, 1000, 1000, 500, 500] (a single distinct vector across all 84,552 frames) — the left hand is unused and held open.
3. 3.Tactile values are byte-swapped. Raw taxels are 12-bit (0–4095) per the recorder schema, but the stored uint16s are little/big-endian flipped: every value's low byte is ≤ 4 and the maximum stored value is 65281. Decoding with ((v & 0xFF) << 8) | (v >> 8) brings the whole dataset into range (observed max 537 left, 507 right). Carried through as-recorded rather than silently fixed; apply decode_taxels above.
4. 4.Neck state is frozen, neck action is not. state[46:48] is 0 for every frame, while action[47:49] varies normally — the neck was commanded but never read back.
5. 5.Single-arm task. Only the right arm manipulates. Left-arm state/action are present and vary slightly (postural), but carry no task signal.
6. 6.All 41 episodes are labelled successful. No negative examples.
7. 7.Body state, action, effort, force, tactile and vision are healthy. All 29 state[0:29] dims, all 5 IMU dims, all 6 root-command dims and all 29 effort dims vary in every episode. All 29 body action dims also vary in every episode (the robot is not standing perfectly still). The usable learning signal is body joints
8. 8.root command + right-thumb command + tactile + effort + force + three-camera vision.

Not carried from the recording

• —Depth. The recording has an empty depth/ directory and every depth field is None — no depth was captured, so there is none to carry.
• —Low-level actions. action_low_level (29-dim motor commands); use --action_mode low_level.
• —Per-frame recorder clocks. t_img / t_action / t_state; timestamp here is derived as frame_index / 60.

Provenance

Single capture session; episode indices map 1:1 to the source episodes:

Reproduce with:

python deploy_real/convert_twist2_to_lerobot.py \
    --data_dir "<recording root>/20260821_1806" \
    --output_dir <out> \
    --repo_id MLeggiero/g1-inspire-nut-in-box-twist2 \
    --fps 60 --action_mode high_level --use_videos \
    --hand_type inspire \
    --include_tactile on --include_effort on --include_force \
    --task_name "pick up the tweezers and put the nut in the box"

The recorder's own info.date is 2026-08-21; its text.goal field still holds the collection stack's placeholder (pick up the red cup) and was not used — the task string above was set explicitly at conversion time. Its info.image block reports width 720 / height 1280, which is transposed; the head frames are 1280×720.

Author: MLeggiero. Data collected on a Unitree G1 + Inspire RH56DFTP teleoperation rig with head + dual wrist cameras, using the open-source TWIST2 collection stack from the YanjieZe repository (tooling credit only). Captured 2026-08-21. Released under the MIT License.