ghkim-rlwrld/shell_game_eef_subtask

1---2license: apache-2.03tags: [robotics, lerobot, real-robot]4---5 6# shell_game — delta EEF, per-frame subtask7 8Cups hide a red block, the cups are shuffled by hand, and the arm lifts the cup the block ended9under. 45 episodes, 18,442 frames.10 11Delta EEF, 10-dim. Six sentences, assigned per frame:12 13```14observe                              6,051 frames15uncover the left cup                 2,20316uncover the middle cup               1,51717uncover the right cup                1,84818move the red cube into the basket    6,82319```20 21Labels are derived from state, never from the video: the gripper separates cup (closure220.120-0.200) from cube (0.440-0.460) with no overlap and exactly two grasps in 45 of 4523episodes, and the cup is read off cartesian y, which is 3-modal. They agree with the human24collection sheet on 44 of 45; the one disagreement is an episode where the operator lifted the25wrong cup.26 27Which cup to uncover cannot be derived from the current frame — the shuffle has already28happened and every cup looks alike. That is the point of the task, and it is why a planner29trained on this needs memory rather than perception.30 31Converted from the recorder's DROID-schema `teleoperation.h5`, one episode per directory, with32the videos re-encoded all-intra. The raw h5 is the source of record and lives at33`ghkim-rlwrld/shell_game`; anything that needs a different action representation or a fresh label34pass has to go back to it, not to this tree.35 36Cameras are three stereo pairs and only the left eye is shipped here, which is what a policy37reading three streams actually loads:38 39```4011845904  wrist        looks down over the gripper4137149196  cam_ext_left   external, to the robot's left4234022131  cam_ext_right  external, to the robot's right43```44 45Control rate is 9.2 Hz measured. The mp4 containers say 10 fps and are wrong.46 47### Naming48 49`<task>_<action space>_<language>`. An earlier set of names took the action space from one50dataset and the language from another, so they could not be compared side by side.51 52| | action | language |53|---|---|---|54| `shell_game_joint_global` | 7 joints + gripper (8) | one sentence per episode |55| `shell_game_eef_global` | delta EEF, xyz + rot6d + gripper (10) | one sentence per episode |56| `shell_game_eef_subtask` | delta EEF (10) | per-frame subtask |57| `solve_equation_eef_subtask` | delta EEF (10) | per-frame subtask |58 59### Why rot6d and not rpy60 61This recorder's roll sits at +-pi and flips sign 269 times across the 45 shell_game episodes.62Regressing the raw angle puts a 2*pi step in the target at every flip. Unwrapping and moving to63the first two columns of the rotation matrix leaves 0 discontinuities and takes the roll channel64from std 3.08 to 0.039. The delta mask is `(9, -1)`: position and rotation relative to the65current state, gripper absolute.66 67A runtime that wants 7-dim xyz+rpy can recover it exactly — Gram-Schmidt the two 3-vectors, take68the third by cross product, then `as_euler("xyz")`, verified to 4.44e-16 over 41 episodes. Doing69that at inference is safe; training on rpy is not, for the reason above.70