DaoyuanZhu/wb_push_cart_stop_go

wbpushcartstopgo

Whole-body teleoperation data from a Unitree_G1_WholeBody_RGB, published in LeRobot v2.1 format.

Published in the v2.1 layout (one parquet and one video clip per episode) so it loads directly on older lerobot releases. On lerobot v3.0+ run the official upgrade first: `` python -m lerobot.datasets.v30.convert_dataset_v21_to_v30 --repo-id=DaoyuanZhu/wb_push_cart_stop_go ``

Task — push the cart, stop when the human raises a hand, and continue when the human waves again

At a glance

Hand state layout

This main revision stores a compact 41-D observation.state: the two unused padding slots from the legacy 43-D layout are removed. The 12 hand values come from the matching real_inspair state, six per hand. action remains the original 43-D command vector.

The matching real_inspair revision retains the original 43-D layout, with six Inspire state values plus one zero-padded slot per hand.

Schema

observation.state is 41-D and action remains 43-D:

 0–11  legs      left/right  hip_pitch, hip_roll, hip_yaw, knee, ankle_pitch, ankle_roll
12–14  waist     yaw, roll, pitch
15–21  left arm  shoulder_pitch/roll/yaw, elbow, wrist_roll/pitch/yaw
22–27  left hand pinky, ring, middle, index, thumb_pitch, thumb_yaw
28–34  right arm shoulder_pitch/roll/yaw, elbow, wrist_roll/pitch/yaw
35–40  right hand pinky, ring, middle, index, thumb_pitch, thumb_yaw

In real_inspair, the Inspire hand reports a 6-D estimate: slots 22–27 / 36–41 carry values and slot 28 / 42 stays zero-padded. On the action side the same slots carry the native command and the last slot is a binary trigger (0.0 / 1.0).

The hand state on real_inspair is estimated, transferred from captures that did record Inspire feedback — the legacy rig had no hand sensing. Describe it as estimated/transferred Inspire hand state, not as recorded ground truth.

Teleoperation stream

Beyond the 41-D state and 43-D action, every frame carries the raw teleoperation and controller signals the capture recorded:

Video

All video is H.264 and plays in any standard player.

Pose annotations

Keypoints ride along as parquet columns, one row per frame — no separate annotation files.

Zero-filled coordinates with valid = 0 mean nothing was detected; there are no NaNs.

How the annotations were filtered

Body detections are post-processed per episode: score threshold, duplicate suppression by bounding-box containment (plain IoU misses a sprawling false box that swallows the real one), fragment merging, then a single-identity lock built from motion tracklets merged on a saturation-masked torso-hue histogram. Tracklets that overlap in time are never merged, since co-visible detections cannot be the same person.

Hand candidates are additionally anchored to the central person's wrist — a detection further than half a shoulder-width from that wrist is rejected, which is what keeps hand-shaped props out of the annotations.

Layout

data/chunk-000/episode_000000.parquet             one parquet per episode
videos/chunk-000/<video_key>/episode_000000.mp4   one clip per episode
meta/info.json                                    feature schema
meta/episodes.jsonl                               episode index and lengths
meta/tasks.jsonl                                  task strings
meta/stats.json                                   per-feature statistics

Loading

from lerobot.common.datasets.lerobot_dataset import LeRobotDataset

ds = LeRobotDataset("DaoyuanZhu/wb_push_cart_stop_go", revision="real_inspair")
print(ds.meta.features.keys())
sample = ds[0]

License

Apache-2.0.