mickeykang/dvla-can2k-futee-sweep

can2k — action-label lookahead sweep

Four LeRobot datasets built from the same 2,005 MuJoCo/robosuite demonstrations. The images, states and episode boundaries are identical across all four; only the action label differs, by the size of the future-EE relabel window (make_lerobot --futee_offset).

The 320 ms variant this sweep was built to test against lives in its own repo: `dvla-can2k-fixed-250hz-events-250fps-delta-trim`.

Task

place: pick a can off the table and drop it into a bowl. 80% of episodes start with the can rolling (speed drawn uniformly from 0.25–1.5 m/s); the rest are static. Demonstrations come from a scripted state machine with privileged simulator state; only successful attempts are kept. Single object (can11), five receptacles. Control frequency 250 Hz, Panda arm under a differential IK + stiff joint-position controller (IK_KP=2500, whose measured end-effector speed ceiling is 1.11 m/s — comparable to the fastest cans, so the arm intercepts rather than chases).

Layout

LeRobot v2.1, robot_type: panda, 3 RGB cameras at 360×480 (wrist_cam, side_cam, opst_cam) plus a v2e event-polarity rendering of each. Training uses opst_cam + wrist_cam.

• —action (10,) = [dx, dy, dz, sin/cos of the 3 euler angles, gripper]
• —observation.state (9,) = [x, y, z, sin/cos of the 3 euler angles]
• —observation.environment_state (9,) = privileged object pose/velocity; drop it for training.

action is a delta: the absolute target pose minus the state of that same frame. At evaluation the policy output is decoded as target = live_state + predicted_delta, re-anchored every step.

meta/camera.jsonl maps each episode_index back to its source HDF5 filename. Conversion shards round-robin, so episode order is not source order and this file is the only way back.

Why the lookahead matters

The offset chooses what the absolute target means. At futee0 the label is the pose the state machine commanded at time t — the controller's actual tracking error. At futee>0 it is the pose the arm had reached N frames later, i.e. a displacement over that window.

The evaluation loop feeds the decoded target to the controller as an immediate set-point every 4 ms, so a lookahead label is consumed as if it were tracking error. Measured on 18 held-out scenes, replaying the labels themselves (the ceiling any policy trained on them could reach):

Too large and the arm is told to close a 11 cm gap in one 4 ms tick: it runs the demonstrated path several times too fast and overshoots (traced on a static scene, the arm reaches the expert's frame 556 by replay step 100, then oscillates 20–32 cm away). Too small and it crawls and runs out of time. Only futee0 reproduces the demonstrations, which is why it is the recommended variant here. Replaying the source HDF5's absolute state-machine actions on the same scenes succeeds 12/12, so the scenes, physics and success criterion are not the limiting factor.

A training-free learnability check points the same way: among frames from different episodes whose physical situation is nearly identical, the spread of the label (relative to its overall spread) is 0.001 at futee0 and rises to 0.178 at futee20 and 0.237 at 320 ms — a longer window makes the label depend on future physics the observation cannot reveal.

Note that these numbers are specific to 250 Hz control with this controller. At 25 fps a 200 ms lookahead happens to produce a commanded step of 4.0 cm against a 4.1 cm tracking error (0.97×), which is why the 25 fps recipes used --futee_offset 5 without trouble.

Reproducing

Generation, conversion, training and evaluation scripts are at <https://github.com/mickeykang16/DynamicVLA/tree/mujoco>.