cloudwalk-research/psi0-g1-he-pickplace-9x40ep

psi0-g1-he-pickplace-9x40ep

9 curated pick-and-place tasks from the [Humanoid-Everyday](https://huggingface.co/datasets/USC-PSI-Lab/humanoid-everyday) (HE) dataset, re-targeted from the Unitree G1 + Dex3-1 3-finger hands onto our G1 + Inspire RH56DFTP 6-DoF lockstep schema — so it can be mixed directly into LeRobot v2.1 fine-tunes of Psi0 (or any policy trained against the Inspire convention) without further preprocessing.

v1.3 update (2026-05-15) — fixes a RH-only polarity inversion that v1.0/v1.1/v1.2 all carried silently. The Dex3-1 right-hand has a mirror-opposite joint sign convention vs the left hand (positive flex = extension for LH, but = flexion for RH), so the same (flex_sum − q05) / (q95 − q05) formula produced correct labels for LH but inverted labels for RH. v1.1's "global polarity fix" was empirically verified against an LH-active task only; the 4 RH-active tasks (place_a_cube_in_a_bag, stack_two_boxes, stack_two_cubes_g1, take_out_the_lid_of_the_spray_bottle) shipped with inverted hand commands until 2026-05-15 real-robot replay surfaced the perfect LH-OK / RH-inverted split. Fix: hand_open_scalar() now applies closure = 1 − closure for hand_side="rh" after the q05/q95 normalization — single point of inversion, propagates to both state and action channels uniformly. If you cloned an earlier snapshot, re-download: the 4 RH-active task subfolders' parquet files have been regenerated. v1.2 update (2026-05-14) — the active-hand action channel is now binarized to `{0, 1}` via a causal Schmitt-trigger hysteresis (bias=0.6, delta=0.10), applied per-episode on top of the v1.1 closure scalar. Plateau jitter and brief multi-frame transition inversions observed during real-robot replay are eliminated. The state channel is unchanged — it still records the raw closure scalar for truthful observation. See §"Hand conversion explained → (c) Per-task normalization" and the conversion_report.md "Hand-action temporal cleanup" section for details. If you cloned an earlier snapshot, re-download.

If you are building VLA / imitation-learning policies on humanoids whose end-effector is driven as a single open/close scalar broadcast to 6 channels (Inspire-class hands, or any "lockstep-trigger" teleop), the upstream HE Dex3-1 recordings are useful task variety but the per-finger 7-DoF action schema is incompatible. This dataset is the bridge: same upstream visual / language / kinematic content, repackaged into the canonical 36-D action / 32-D state layout used by our cloudwalk-research/psi0-g1-sneaker-* family, so a single compose_lerobot_group.py call mixes it cleanly with native Inspire teleop data.

TL;DR

What's in the box (9 sibling task subfolders)

Each subfolder is a self-contained LeRobot v2.1 dataset with its own data/, videos/, meta/, and a .conversion_summary.json audit file.

(A 10th candidate task — put_can_of_chips_on_drawer — was selected but is absent from HE_RAW_no_static.zip upstream, so this release ships 9 of 10.)

Why this dataset exists

Humanoid-Everyday recorded its G1 episodes with Dex3-1 3-finger hands: thumb (3 DoF), index (2 DoF), middle (2 DoF) = 7 articulated DoF per hand. Our deployment hardware uses Inspire RH56DFTP 5-finger hands, but our teleop stack drives them in lockstep — a single VR-trigger value broadcast to all 6 actuators per hand. So the policy's hand-action schema is one scalar in [0, 1] (1.0 = fully open, 0.0 = fully closed), repeated 6× into action[0:6] for LH (or [7:13] for RH), with a zero pad at [6] / [13].

That means consuming HE data needs a 7 → 1 projection that preserves the temporal grasp pattern (the moments where the hand visibly closes around an object and later releases) — without inventing per-finger nuance our hardware can't actuate.

This dataset is one rigorous and validated implementation of that projection. Everything below explains how and gives the empirical evidence that it's correct.

Hand conversion explained

(a) The two hands, side by side

• —Inspire (target hardware). 5 fingers, 6 actuators, driven by one open/close scalar broadcast to 6 channels per hand (deliberate; see the v4 sneaker reference dataset cloudwalk-research/psi0-g1-sneaker-in-box, whose meta/stats_psi0.json shows the idle LH pinned at mean=1.0, std=0 and the active RH spanning mean=0.574, std=0.482, min=0, max=1 — full bimodal occupancy).
• —Dex3-1 (source hardware). 3 fingers, 7 joints per hand, each one is a real angular DoF in radians. The 7→1 projection has to map these 7 numbers onto one scalar per frame, keeping the temporal sequence of open → reach → grasp → transport → release → open intact.

(b) The trick — sum of four "tip flexion" joints

Pick the 4 joint indices that always represent fingertip flexion regardless of LH / RH ordering: [1, 2, 4, 6].

LH joint order: [thumb_0, thumb_1, thumb_2, middle_0, middle_1, index_0,  index_1 ]
RH joint order: [thumb_0, thumb_1, thumb_2, index_0,  index_1,  middle_0, middle_1]
                          ^^^^^^^  ^^^^^^^            ^^^^^^^             ^^^^^^^^
                          [1]      [2]                [4]                 [6]

• —On LH, [1, 2, 4, 6] = thumb_1, thumb_2, middle_1, index_1 — distal flexion of all three fingers.
• —On RH, [1, 2, 4, 6] = thumb_1, thumb_2, index_1, middle_1 — distal flexion of all three fingers (same physical role; middle ↔ index swap in slots 4 and 6).

We sum the four values. Sum is commutative, so the slot swap is invisible — one formula works on both hands. We use distal joints rather than proximal because distals have larger angular swing (~90° vs ~30° for proximals) and dominate the "is the hand visibly curled" signal.

(c) Per-task normalization (q05 / q95) — why not URDF joint limits?

For each task, compute the 5th and 95th percentile of that 4-joint sum over all 40 episodes' commanded actions (frame["actions"]["left_angles"] / right_angles). Then linearly remap:

open_scalar = clip((flex_sum - q05) / (q95 - q05), 0, 1)

Plain-language: the most-open state this task ever reaches becomes 1.0; the most-closed state becomes 0.0. Every task spans the full output range.

Polarity note (v1.1 LH, v1.3 RH). An earlier v1.0 release had a 1 − closure term in the formula above, written under an empirically wrong assumption about the Dex3-1 distal-flex sign convention. Real-robot replay on G1+Inspire surfaced the inversion (scalars opened on grasp and closed on release). v1.1 dropped the 1 − closure and was verified against unstack_two_cubes_g1 (LH-active), confirming LH joints read POSITIVE on extension (opening) and NEGATIVE on flexion (closing). The 4 RH-active tasks were not directly validated post-fix and silently shipped inverted across v1.1 and v1.2; full 9-task real-robot replay on 2026-05-15 surfaced a clean LH-OK / RH-inverted split, pinning the root cause to a mirror-opposite joint sign convention on the Dex3-1 RH (the SDK calibrates LH and RH so that a positive joint command produces inward/closing motion on both hands, which means the angle axes are mirrored, so raw joint values have opposite signs for the same physical configuration). v1.3 adds a hand_side kwarg to hand_open_scalar() and applies closure = 1 − closure for hand_side="rh" only — both state and action channels go through the same function so the fix is uniform. If you cloned an earlier snapshot of this repo, re-download — the 4 RH-active task subfolders' parquet files have been regenerated.

Hysteresis note (v1.2, 2026-05-14). The formula above produces the state channel. For the action channel we add a causal Schmitt-trigger binarize on top, per-episode and per-active-hand:

lower = bias - delta / 2    # 0.55 with defaults bias=0.6, delta=0.10
upper = bias + delta / 2    # 0.65 with defaults
state = 1 if scalar[0] >= bias else 0
for x in scalar:
    if state == 1 and x < lower: state = 0
    elif state == 0 and x > upper: state = 1
    out.append(float(state))

Output is 0.0 or 1.0 only. Two real-robot failure modes motivated this: (1) plateau jitter — the raw scalar bobbed in [0, 0.15] / [0.85, 1.0] during "fully closed" / "fully open" plateaus; (2) transition inversion — the 4-joint distal-flex sum is a NON-monotonic proxy and can briefly reverse for 2–5 frames during a physical closure as individual fingers flex out-of-phase. The asymmetric bias (0.6) is closed-friendly — partial closures down to ~45 % trigger the closed state. The state channel is left untouched so policies can still see the raw observation. An optional --hand-prefilter median3|ema|none is wired in the converter for future tuning; this release ships with prefilter=none.

Why not URDF joint limits? Because the absolute range a task uses depends on the task. A delicate insertion task might only flex through ~40 % of the hand's physical range; mapping to URDF limits would squash its grasp/release signal into a narrow band. Per-task normalization restores the bimodal shape uniformly.

Trade-off. scalar = 0.5 does NOT mean the same absolute Dex3 angle across tasks. For deployment on Inspire that is fine — the hardware just sees a scalar trajectory. Absolute grip-depth calibration in your downstream policy should come from a native-Inspire dataset (e.g., our v4 sneaker), not from this rehearsal slice alone.

(d) Idle-hand pinning to 1.0

In a unimanual task, the resting hand's Dex3 joints still drift through 1–3 rad due to whole-body IK without ever touching anything. We pin that hand's scalar to 1.0 (fully open) for the entire task. This matches the safe-default Inspire convention and reproduces the v4 sneaker convention byte-for-byte (the sneaker is a RH-only task; v4 stats show idle LH mean=1.0, std=0.0 across all 385 episodes).

Active-hand assignment is hard-coded per task via the converter's --active-hand {lh,rh,both} flag. Range-based auto-detection was tried and discarded — idle whole-body-IK drift can exceed the active grasp range, so the heuristic mis-classified.

(e) Lockstep broadcast + zero pads

The single scalar is copied to all 6 channels of its hand block (action[0:6] for LH, action[7:13] for RH; same for state at indices [0:6] / [7:13]). Pads at index [6] and [13] are zero. This layout is byte-identical to psi0-g1-sneaker-in-box, required for compose_lerobot_group.py::require_compatible_sources to accept the dataset for fine-tune composition.

What guarantees correctness?

Each line below is one consistency guarantee plus the artifact that proves it.

• —Temporal alignment is preserved by construction. Per-frame deterministic remapping: frame t of the source Dex3 episode → one scalar at the same frame_index = t in our parquet. Frame count, FPS (30), and frame_index sequencing are byte-equal to the source episode.
• —Same calibration for state and action. task_closure_bounds() is computed once per task from the action distribution, and both build_state() and build_action() use the same (q05, q95). The state channel reports that scalar as-is; the action channel passes it through the v1.2 Schmitt-trigger binarize. The state-mirror invariant (state[t, 0:6] ≈ action[t-1, 0:6]) therefore only holds at the plateaus (state near 0 ⇒ action 0; state near 1 ⇒ action 1); during transitions the two diverge by design. Catching wrong-hand-assignment bugs has been moved to the per-task median-per-episode q95 − q05 ratio at conversion time ([WARN] fires at ratio < 0.7; correctly-labeled tasks have margins of 3.5–26.8×).
• —Direction (open vs closed) is empirically confirmed via real-robot replay. For every active hand, q05 < 0 < q95 in radians. Examples:
• —unstack_two_cubes_g1 LH (v1.1, relabeled): q05 = -1.667, q95 = +1.453 (range 3.12 rad)
• —place_a_cube_in_a_bag RH: q05 = -1.483, q95 = +1.708 (range 3.19 rad)
• —pick_up_a_bottle_and_hand_it LH: q05 = -1.589, q95 = +1.214 (range 2.80 rad)

LH Dex3-1 distal-flex joints read POSITIVE on finger extension (opening) and NEGATIVE on flexion (closing). RH Dex3-1 distal-flex joints have the OPPOSITE sign convention (positive on flexion, negative on extension) — this is a SDK-level joint-axis mirror, not a data bug. v1.3 handles it by applying closure = 1 − closure inside hand_open_scalar() whenever hand_side="rh", after the per-task q05/q95 normalization. Plug in: for LH, flex = q05 (closed) → open_scalar = 0.0 ✓, flex = q95 (open) → open_scalar = 1.0 ✓. For RH, flex = q95 (which is the physically-CLOSED state because of the sign mirror) → raw closure = 1.0 → 1.3 inversion → open_scalar = 0.0 ✓; flex = q05 → raw closure = 0.0 → inverted → open_scalar = 1.0 ✓.

• —Distribution is bimodal — open AND closed phases both well-represented. Active-hand action (v1.2+) is binarized: every frame is either 0.0 (closed) or 1.0 (open), with closed % + open % ≈ 100 %. Per-task active-hand closed % ranges from 35 % to 72 % and open % fills the remainder — every task has both phases meaningfully represented. The state channel keeps the raw scalar (mid-range mass is intact for observation).
• —Idle-hand handling matches the v4 reference byte-for-byte. v4 sneaker idle LH: mean=1.0, std=0.0, min=max=1.0. Our 5 LH-active tasks all show mean=1.0, std=0.0 on the idle RH; our 4 RH-active tasks the same on the idle LH. The pin-to-1.0 rule reproduces the v4 ground truth exactly.
• —Schema parity is byte-equal. info.json::features matches v4 on every byte for all 9 subfolders. Lockstep, pad, range, episode-count, next.done, and frame_index gates all pass.
• —Information-loss budget is bounded. Frames where Dex3 commanded actions were missing would fall back to next-state-as-action; observed fallback fraction is 0 % across all 9 tasks — every HERAW frame had complete `actions.leftangles / rightangles / solq`.

Documented assumptions

What we're trusting that could in principle be wrong, and how to catch each one.

• —Dex3-1 LH and RH have OPPOSITE distal-flex sign conventions (corrected for LH in v1.1, for RH in v1.3). LH joints [1, 2, 4, 6] read positive on finger extension (opening) and negative on flexion (closing); RH joints [1, 2, 4, 6] are mirrored — positive on flexion, negative on extension. The converter applies closure = 1 − closure to RH non-idle frames inside hand_open_scalar() to compensate. Both bug versions (LH inversion in v1.0, RH inversion in v1.0–v1.2) shipped silently because a uniform sign flip is NOT caught by the validation gates: q05 ↔ q95 swap symmetrically, the bimodal distribution still passes the open/closed quota, and schema parity is byte-equal regardless. The only robust self-detection is a real-robot replay or a visual scrub — play one episode's .mp4 alongside the parquet scalar trace and confirm closure → 0.0 precisely at the visible grasp moment for both LH-active and RH-active tasks.
• —Per-task normalization is appropriate only when this dataset is mixed with absolute-calibration data. HE provides task variety; a native-Inspire dataset (e.g. psi0-g1-sneaker-in-box) provides absolute grip-depth calibration. Training a policy only on HE-derived data would inherit the per-task scale and likely under- or over-grip at deployment.
• —Active-hand assignment is hard-coded per task (see the per-task active-hand column above). A wrong assignment pins the actual active hand to constant 1.0 (silently drops the grasp signal) AND modulates the actual idle hand with whole-body-IK drift that's q05/q95-normalized into a quasi-random high-frequency signal — both visible on the real robot. The original v1.0 release mislabeled unstack_two_cubes_g1 for this reason; v1.1 fixes the label and adds a [WARN] at conversion time when the labeled active hand has a meaningfully smaller commanded range than the idle hand.
• —v1.2 hysteresis assumes a real partial-grip plateau is rare AND deep enough to cross the lower threshold (~0.55). The asymmetric bias=0.6, delta=0.10 is closed-friendly: any closure that drops the per-task normalized scalar below 0.55 (i.e. crosses below ~45 % of the open→closed dynamic range) latches to 0.0. A grip on an unusually large object that only crosses, say, 30 % might never trip the lower threshold and stay reported as OPEN on the robot. Self-detection: regenerate the _v1.3_diagnostic.png plot (raw vs. binarized scalar, episode 0 of 3 representative tasks) and inspect whether legitimate grip plateaus get binarized correctly. Mitigation paths: (a) lower --hand-hysteresis-bias toward 0.5 for that task only; (b) enable a small pre-filter with --hand-prefilter median3; (c) disable binarize entirely with --no-hand-hysteresis to fall back to v1.1 raw-scalar behavior.

Schema and layout (action 36-D / state 32-D)

action[0:6]    LH hand — lockstep open/close scalar broadcast to 6 channels, range [0, 1]
action[6]      LH pad — constant 0.0
action[7:13]   RH hand — lockstep, broadcast 6×, range [0, 1]
action[13]     RH pad — constant 0.0
action[14:21]  LH arm  — 7-DoF G1 joint angles, radians
action[21:28]  RH arm  — 7-DoF G1 joint angles, radians
action[28:31]  Torso RPY — constant (0, 0, 0)
action[31]     Torso height — constant 0.75
action[32:36]  Locomotion (vx, vy, vyaw, target_yaw) — constant (0, 0, 0, 0)

state is the same except it omits action[32:36] (32-D total).

Per-task validation summary

Closure stats and gate outcomes for each task's active hand. All 9 datasets passed every validation gate. The closed % / open % columns are post-v1.2 binarize, so they sum to 100% (no mid-range mass on the action channel).

(Action-channel closed % = fraction of frames at exactly 0.0; open % = fraction at exactly 1.0. Sum ≈ 100% for all active hands by construction of the Schmitt-trigger binarize. Idle-hand columns omitted — they are identically 1.0 by design and are reported separately in each .conversion_summary.json. v1.3 changes from v1.2: the 5 LH-active rows are byte-identical (LH untouched by the v1.3 fix); the 4 RH-active rows have closed/open swapped relative to v1.2, with an additional 10–20 pp shift toward CLOSED because the closed-friendly Schmitt bias (0.6) now correctly catches mid-range partial-grip frames as closed instead of as open.)

How to use

Download the whole pool with one call:

import os
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="cloudwalk-research/psi0-g1-he-pickplace-9x40ep",
    repo_type="dataset",
    local_dir="data/he_lerobot",
    token=os.environ.get("HF_TOKEN"),  # private; set HF_TOKEN
)

Each subfolder is a valid LeRobot v2.1 dataset on its own — load it directly:

from lerobot.common.datasets.lerobot_dataset import LeRobotDataset

ds = LeRobotDataset("data/he_lerobot/unstack_two_cubes_g1")
print(ds[0]["action"].shape)   # torch.Size([36])
print(ds[0]["states"].shape)   # torch.Size([32])

Or compose any subset with a native-Inspire dataset (e.g., the v4 sneaker) for a mixed fine-tune corpus:

python scripts/data/compose_lerobot_group.py \
    --datasets   data/real/psi0-g1-sneaker-385ep-v4 \
                 data/he_lerobot/unstack_two_cubes_g1 \
                 data/he_lerobot/put_dumpling_into_plate_g1 \
    --output-name psi0-g1-sneaker-he-mix-461ep-v5 \
    --root-dir   data/he_lerobot \
    --video-mode hardlink

Related datasets and models

Provenance

• —Source: USC-PSI-Lab/psi-data → HE_RAW_no_static.zip (176 338 894 892 bytes verified at conversion time).
• —Converter: he_raw_to_psi0_inspire.py (CloudWalk Robotics Lab internal tooling).
• —Per-task active-hand assignment: sourced from a curated 10-task spreadsheet maintained internally; the 9 included tasks are the unimanual pick-and-place subset that survived upstream availability + validation gates.
• —Schema-parity reference: cloudwalk-research/psi0-g1-sneaker-in-box (meta/info.json and meta/stats_psi0.json).
• —Conversion report: conversion_report.md at this repo's root carries the full audit trail (per-task closure stats, threshold tuning rationale, compose smoke-test outcome, v1.2 hysteresis flip-count drop table, v1.3 RH polarity-fix retrospective).
• —Per-task audit file: <task_slug>/.conversion_summary.json records the closure bounds, validation outcome, and (v1.2+) a hysteresis block listing the params used (enabled, bias, delta, prefilter) plus the per-task flip-count before/after the Schmitt-trigger binarize.
• —Diagnostic plot (local-only): the conversion pipeline can re-emit _v1.3_diagnostic.png showing raw vs. binarized scalar for one episode of three representative tasks. Not shipped in this repo (debug artifact); see conversion_report.md for the per-task flip-count summary instead.

Attribution

Developed by cloudwalk-research in the CloudWalk Robotics Lab (CW-RL). Part of the Psi0 collection. Built to extend the Psi0 base model training distribution with HE-derived pick-and-place tasks, re-targeted to the G1+Inspire lockstep schema and stored in the LeRobot v2.1 format.

License & acknowledgements

This dataset is a non-commercial derivative of the upstream Humanoid-Everyday release by USC-PSI-Lab. The license is inherited from upstream (see `license_link`) — please review and respect the upstream terms when using this adaptation. The remapping into the Inspire 6-DoF lockstep schema, validation tooling, and packaging are released as-is for research use.

Thanks to:

• —The Humanoid-Everyday team at USC for releasing the raw recordings as open data, complete with frame-level commanded actions (actions.left_angles / right_angles / sol_q) — without which a faithful 7→1 projection would not be possible.
• —The LeRobot maintainers at Hugging Face for the v2.1 schema and the framework that made the downstream composition trivial.
• —The Physical Superintelligence Lab for the Psi0 model and open documentation, which defines the action / state layout this dataset is shaped to.

References

• —Humanoid-Everyday primary release — `USC-PSI-Lab/humanoid-everyday` (LeRobot v2.1, 8 949 ep / 246 tasks).
• —Humanoid-Everyday raw source — `USC-PSI-Lab/psi-data` → HE_RAW_no_static.zip (carries frame-level commanded actions; used as the source-of-truth for this adaptation).
• —Psi0 — generalist humanoid VLA model. Paper (arXiv:2603.12263) · GitHub.
• —LeRobot — the v2.1 schema and storage framework used by this dataset. GitHub.
• —Unitree G1 — humanoid robot platform recorded in both upstream HE and our native datasets. Product page.
• —Unitree Dex3-1 — 3-finger end-effector used for the upstream HE recordings.
• —Inspire RH56DFTP — 5-finger end-effector targeted by this adaptation, driven by our teleop stack in a single lockstep open/close channel.

Citation

If you use this dataset, please cite both the upstream Humanoid-Everyday release and this adaptation. If you fine-tune Psi0 on top, please also cite the Psi0 paper.

@misc{cloudwalk_psi0_g1_he_pickplace_2026,
  title  = {psi0-g1-he-pickplace-9x40ep --- Humanoid-Everyday G1 pick-and-place tasks re-targeted to G1+Inspire lockstep schema},
  author = {CloudWalk Research},
  year   = {2026},
  howpublished = {Hugging Face dataset},
  url    = {https://huggingface.co/datasets/cloudwalk-research/psi0-g1-he-pickplace-9x40ep},
}

@misc{humanoid_everyday_2025,
  title        = {Humanoid-Everyday: a multimodal dataset for humanoid manipulation},
  author       = {USC Physical Superintelligence Lab},
  year         = {2025},
  howpublished = {Hugging Face dataset},
  url          = {https://huggingface.co/datasets/USC-PSI-Lab/humanoid-everyday},
}

@article{psi0_2026,
  title         = {Psi0: A Generalist Vision-Language-Action Model for Humanoid Robots},
  author        = {Wei, Songlin and others},
  year          = {2026},
  eprint        = {2603.12263},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2603.12263}
}

@article{black2024pi0,
  title   = {{$\pi_0$}: A Vision-Language-Action Flow Model for General Robot Control},
  author  = {Black, Kevin and Brown, Noah and Driess, Danny and others},
  journal = {arXiv preprint arXiv:2410.24164},
  year    = {2024}
}

@software{lerobot2024,
  title  = {{LeRobot}: State-of-the-art Machine Learning for Real-World Robotics in {PyTorch}},
  author = {{Hugging Face}},
  url    = {https://github.com/huggingface/lerobot},
  year   = {2024}
}