OOJU/unitree-g1-pick-place-cup

OOJU Unitree G1 Pick and Place Cup

47 cup pick-and-place recordings on a Unitree G1, shipped with trained policy checkpoints and the evaluation results for each of them. 38 of those recordings were segmented into 78 first-grade lift candidates, which are the pool every policy below draws from. At that scale a learned diffusion policy reaches 46.7% success over 30 rollouts, while retrieving and replaying the nearest recorded candidate reaches 82% over 11 trials. All evaluation is in Isaac simulation, not on real hardware.

That gap is the point of this release. The recorded motions are good enough that replaying the closest one usually works; there are not enough distinct ones for a policy to learn the mapping and generalise. If you are deciding how much data to collect, the bottleneck this dataset shows is coverage, not demonstration quality.

OOJU dataset series

Series overview with hand-to-robot retargeting accuracy measurements: https://ooju.world/datasets.html

Policy evaluation

The rows do not all use the same training set, and the difference matters when comparing them. The diffusion policy row used all 78 first-grade candidates. The two 11-rollout rows used cluster 1 only: 57 candidates from 14 recordings, grouped by trajectory similarity into the dominant centre-front grasp style, then split 46 train and 11 held out at seed 42. Every row shares the same success criterion: the cup lifted at least 0.10 m and held for at least 15 frames.

An earlier diffusion-policy run scored 10% on 20 rollouts, but it used a 0.01 m lift threshold rather than 0.10 m, so it is not comparable to the rows above and is left out. Its raw output is still in pipeline_output/eval_results/ as dp_eval_v2.json.

Read the two headline numbers with their limits

46.7% (14 of 30), diffusion policy. Pure policy rollouts: no demonstration replay, no scripted lift, no warmup. Successful lifts ranged 13.4 to 19.7 cm, mean 16.4 cm, sustained a mean of 709 frames. This is the number we would ask you to compare against.

81.8% (9 of 11), CVAE with nearest-train latent selection. Four limits worth knowing before quoting it. The trial count is 11, so the interval around 82% is wide. The 11 are a held-out split of one trajectory cluster, not of the whole dataset, so they share a grasp style with what the model trained on. The split is labelled holdout, but 5 of the 11 trials retrieved the exact episode they were tested on, at the floor nearest-neighbour distance of 0.0025. And the 11 rows cover only 8 distinct episodes. It is a useful upper bound on what the recordings contain; it is not a generalisation result.

Reproducing the 46.7%

python pipeline/training/evaluate.py \
  --policy pipeline_output/checkpoints/dp/best.pt \
  --policy-type dp \
  --dataset pipeline_output/lerobot_dataset/ \
  --num-episodes 30 --max-steps 800 \
  --lift-threshold 0.10 --sustained-frames 15 \
  --seed 42 \
  --static-obs --use-action-obs \
  --action-horizon 8 --step-repeats 4 \
  --headless

Six bugs that made a trained policy look broken

Before the 46.7% run, the same checkpoint scored near zero. Two things were wrong. The local checkpoint was undertrained at epoch 433 (valloss 0.0066) against the full run's epoch 1949 (valloss 0.00086), a 7.7x improvement. And the evaluation harness had six observation and control bugs. Every one is a mismatch between how the data was recorded and how the policy was rolled out, which is the usual failure mode after retargeting human demonstrations to a robot.

If you are evaluating a retargeted policy in simulation and it looks broken, this list is the first place to check. Full notes are in pipeline_output/eval_results/eval_notes.md.

Snapshot

Nine of the 47 raw recordings are not represented in the training export. They are still in the repository root; the pipeline did not carry them through segmentation.

Loading

from datasets import load_dataset

ds = load_dataset("OOJU/unitree-g1-pick-place-cup", split="train")

This returns 1,057 rows, not 47. Each row is a candidate lift segment cut out of a raw recording, carrying source_trajectory, candidate_id, grade, max_lift, start_frame and end_frame. Filter to grade == "first" for the 78 segments the published policies use. The parquet features are observation.state and action only, so this export cannot be used to train a visuomotor policy; the rendered videos under pipeline_output/videos/ are separate and are not part of the LeRobot features.

The raw recorder output for each recording sits at the repository root as pick-place-cup_20260404_<time>.json (trajectory), ..._metadata.json and ..._robot.json (robot observations and actions).

Layout

.
├── pick-place-cup_20260404_<time>.json           # raw trajectory  (x47)
├── pick-place-cup_20260404_<time>_metadata.json
├── pick-place-cup_20260404_<time>_robot.json
└── pipeline_output/
    ├── lerobot_dataset/    # LeRobot parquet, what load_dataset reads
    ├── checkpoints/        # trained policy weights
    ├── eval_results/       # per-method metrics and notes
    ├── videos/             # rendered episode videos
    ├── candidates/         # pipeline intermediate
    ├── recordings/         # pipeline intermediate
    ├── results/            # pipeline intermediate
    └── segmentations/      # pipeline intermediate

The four directories marked as pipeline intermediates hold working files from the segmentation stage. Nothing in this card refers to them and they are scheduled for removal; do not build against their paths.

Limitations

• —One task, one object, one robot embodiment, one physical setting.
• —All evaluation is in simulation. No real-hardware success rates are published here.
• —47 demonstrations is small. The evaluation table is there so you can see what that buys rather than having to find out yourself.
• —The two 11-rollout rows have wide intervals. Treat the 20 and 30 rollout rows as the more reliable ones.

FAQ

How many demonstrations do I need to train a manipulation policy? For this task, 78 candidate segments cut from 38 recordings got a learned diffusion policy to 46.7% in simulation. That is enough to show the task is learnable and not enough to deploy. The retrieval baseline at 82% suggests the recorded motions themselves are fine, so the next thing to add is variation in object placement and scene, not more repeats of the same setup.

Why do policies fail after retargeting human demonstrations to a robot? In our experience the failure is usually not the retargeting itself but the mismatch between recording conditions and rollout conditions. The six bugs listed above are concrete instances: index conventions, action chunking, static versus live observations, commanded versus measured state, physics step ratios, and first-frame initialisation.

Can I collect this kind of data without a teleoperation rig? Yes. These demonstrations were recorded with a Meta Quest 3 and hand tracking, then retargeted. See ooju-galaxea-pick-cube for a larger set with 80,014 robot samples.

Can I use one human recording on more than one robot? The pipeline supports retargeting to Unitree G1, UR5e and Galaxea R1 Pro. This repository ships the G1 result; the Galaxea result is a separate dataset. We have not yet published a single session retargeted to two robots side by side.

Collect and validate your own data

[Request access](https://ooju-app.world/request?utm_source=huggingface&utm_medium=dataset_card&utm_campaign=unitree-g1-pick-place-cup&utm_content=footer)

License

Released under CC BY-NC 4.0. Non-commercial use only.

Citation

@misc{ooju_unitree_g1_pick_place_cup,
  title={OOJU Unitree G1 Pick and Place Cup},
  author={OOJU},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/datasets/OOJU/unitree-g1-pick-place-cup}}
}