Harrysunshine/so101-sim-pickplace
SO-101 Sim Pick-and-Place Simulated pick-and-place demonstrations for the SO-101 arm, generated in ManiSkill3 / SAPIEN with a robot model, object geometry and camera calibration matched to a real desktop rig. Config Task prompt Object Episodes Frames Trajectories from cube40 Pick up a cube and place in the bin 40 mm cube 465 172,869 scripted servo cube20 Pick up a small cube and place in the bin 20 mm cube 477 186,438 scripted servo cylinder40 Pick up a can and… See the full description on the dataset page: https://huggingface.co/datasets/Harrysunshine/so101-sim-pickplace.
SO-101 Sim Pick-and-Place
Simulated pick-and-place demonstrations for the SO-101 arm, generated in ManiSkill3 / SAPIEN with a robot model, object geometry and camera calibration matched to a real desktop rig.
The prompts are byte-identical to the corresponding tasks in our real-robot recordings — same object, same physical kit, same wording. A VLA keys on the prompt text, so a single differing word would make it learn one task as two and quietly forfeit the cross-domain transfer. cube20 has no real-robot counterpart; it is worded to sit alongside the others.
★ Units: the same mixed convention a real SO-101 records
action and observation.state are six-dimensional and not normalised. The convention is mixed, because that is what the real robot produces — lerobot-record drives an SO-101 through so_follower, whose per-joint normalisation modes differ:
The gripper is a percentage, not an angle: so_follower hard-codes it to MotorNormMode.RANGE_0_100 regardless of use_degrees. Matching a simulator to a real arm is therefore a per-channel conversion, not one unit change. The percentage is computed from the gripper joint's own URDF limits (−10° … 100°, so 110° of travel):
percent = (degrees − (−10)) / 110 × 100Cross-check against hardware: the simulated fully-open pose is 44.95° → 49.96 %, and the real arm measures 50.6 % at the same pose.
⚠️ Community SO-101 datasets do not all use this convention — HuggingFace's own lerobot/svla_so101_pickplace is normalised to ±100 (its action touches exactly ±100.00, the signature of a clamp). Check before mixing datasets: the gripper channel is the one that bites, because degrees and percent happen to span the same numeric range here, so a wrong assumption looks plausible and only shows up as "grasping never learns".
action is an absolute joint-position target (not a delta), recorded at control mode pd_joint_pos. Observations are 480×640 RGB from two cameras (top, wrist) at 30 fps, robot_type = so_follower.
Every episode's frame 0 is the real arm's power-on pose [-5.76, -102.68, 92.97, 63.38, -0.53, 1.90] (degrees), identical across all 1,649 episodes.
Verification
Driving the simulator with real-robot actions reproduces the real observations before contact: per-channel correlation 0.991–1.000, residual ≤1.22° (arm) / 0.48 % (gripper), zero out-of-limit actions. Replaying this dataset's actions back into the simulator from each episode's recorded initial scene completes the task and reproduces the recorded observations to within the simulator's own run-to-run non-determinism.
The one systematic difference that remains is a ~2° zero-point offset on elbow_flex between the real arm's calibration midpoint and the URDF zero. That is per-robot calibration, not a convention mismatch, and it is deliberately not compensated in simulation.
Only completed episodes are included
Each episode is judged by the environment's own evaluate() (object inside the bin opening ∧ arm not touching the object ∧ arm static ∧ arm not touching the bin): 1,833 → 1,649, dropping 184 (10.0 %). Of those kept, over 99.8 % have the object in the bin, with a median final offset of 0.04–0.21 of the bin opening's half-width.