Bigenlight/flow_matching_carrot_in_pot_ee

1---2license: cc-by-nc-4.03library_name: lerobot4pipeline_tag: robotics5tags:6  - robotics7  - lerobot8  - flow-matching9  - diffusion-transformer10  - imitation-learning11  - ur7e12  - end-effector13---14 15# Flow-Matching DiT · carrot-in-pot · EEF-delta (state 16 / action 7) — checkpoint 60k16 17Text-conditioned flow-matching policy (`multi_task_dit`, CLIP text encoder, Euler ODE) trained on18the **real** UR7e *"Put carrot in pot"* demonstrations (54 GELLO-teleop takes, 30 fps) in the19**EEF-delta action space** (`eef_delta_v1`). This is the **60k-step checkpoint** (best open-loop20chunk-30 error of 10k..100k; kanu kept only 100k).21 22## Observation / action space (`eef_delta_v1`)23- `observation.state` **16-D** = `[q1..q6 (rad, UR order), tcp_x, tcp_y, tcp_z (m, base_link),24  r11, r21, r31, r12, r22, r32 (first two columns of the TCP rotation), grip_pos (0=open..1=closed)]`.25  TCP = `ur_kin.fk(q)` + 0.174 m along flange +Z.26- `action` **7-D** = `[dx, dy, dz, drx, dry, drz, grip_cmd]` — achieved per-frame TCP motion27  (`dp = p_{t+1}-p_t`, `drot = so3_log(R_{t+1} R_t^T)`, base frame), gripper absolute 0..1.28  Deploy: `p_target = p_live + dp`, `R_target = so3_exp(drot) R_live`, analytic IK, branch locking.29- Cameras: `cam1` (scene), `cam2` (wrist), native 720×1280 fed in; the policy resizes internally30  to `image_resize_shape = [224, 224]` — **do not pre-resize** (double-resize silently degrades).31- Task string: `"Put carrot in pot"` (CLIP-conditioned; send it every tick).32- `chunk_size 32`, `n_action_steps 24`, `n_obs_steps 2`, `num_integration_steps` 100 at train33  time (10 is the usual serving override), DiT hidden 512 × 6 layers.34 35## Training36- Dataset: `carrot_in_pot_eef_lerobot_v3` — local LeRobot v3 re-export of37  [`Bigenlight/carrot_in_pot_lerobot_v3`](https://huggingface.co/datasets/Bigenlight/carrot_in_pot_lerobot_v3)38  (54 ep / 17,085 frames; joints de-lagged by per-take τ≈0.90 s + linear re-interpolation).39  Not yet on the Hub.40- `lerobot-train`, batch 8, seed 1000, 100k steps (10 h 15 m on one RTX A4000, kanu), job `fm_carrot_eef`.41 42## Held-out results (open-loop, chunk-30 cumulative error, episodes 48–53)43| 10k | 20k | 30k | **60k (this)** | 100k | zero-motion |44|---|---|---|---|---|---|45| 34.8 mm | 32.8 mm | ~33 mm | **32.0 mm** | 32.6 mm | 65.6 mm |46 47lerobot's `eval_loss` rose monotonically 0.096→0.233 over the run while the open-loop metric48kept improving — it was ignored for checkpoint selection.49 50## Status51Real-robot closed-loop evaluation: **not yet run**. The shipped ZMQ servers52(`gello_policy/policy_server/fm_server.py`) are joint-space 7/7 and refuse this checkpoint's5316-D state; serving needs the EEF-mode deploy node from branch `feat/carrot-eef-il`.54