Myungkyu/RoboDojo-taco-visual-gemini

RoboDojo-taco-visual-gemini

The visual-grounding variant of RoboDojo-taco-gemini: the same RoboDojo long-horizon episodes (8 tasks, 800 episodes, 25 fps) with the same dense high-level labels (Gemini 3.7 Flash under the task-specific context induced for each task), except that a target position leaves the label text and is drawn into the low-level policy's keyframe slot.

• —In the source labels a target that words cannot identify is named by its image coordinates on the 0–1000 scale of the head camera frame, e.g. Place the left pen at (659, 485) in the pen holder.
• —Here that label reads Place the left pen at the location marked in the keyframe in the pen holder., and the keyframe stream of the episode shows the head camera frame with a point marker (red disk, white ring, radius 1.9 % of the frame width) at (659, 485) for every frame of that label's span.
• —Frames whose label names no position keep a keyframe that is an exact copy of the head camera frame.
• —play_tic_tac_toe names board cells instead of coordinates (… at the back middle.); the board is fixed in the scene, so those labels become Place the first O piece at the back middle marked in the keyframe. and the cell centre is marked.

Layout

<task>/                      LeRobot v2.1 dataset of one task (data/, meta/, videos/), episodes re-indexed 0..99
  videos/chunk-000/observation.image.head_camera/     head camera (RoboDojo cam_high, 640x480, 25 fps)
  videos/chunk-000/observation.image.left_wrist/      left wrist camera (cam_left_wrist)
  videos/chunk-000/observation.image.right_wrist/     right wrist camera (cam_right_wrist)
  videos/chunk-000/observation.image.keyframe/        head camera copy with the point marker of the current label (declared as a video feature)
  meta/dense_annotation_map.json                      per-episode subtask spans, `marked_frames` per episode
  meta/robodojo_reindex.json                          RoboDojo episode index -> this dataset's index
retrieval_map.json           per task and episode (no retrievals in this benchmark)
annotations/<task>/ep%06d.jsonl   the raw per-tick labels: `subtask` (visual wording), `subtask_source` (the coordinate wording),
                                  `marker_xy` (the marked points, 0–1000 scale), `updated_memory`

Per-frame features: observation.state (14: left arm 6 joints + gripper, right arm 6 joints + gripper), action (14), subtask, global_task, subtask_end, episode_id, the three camera views and the keyframe slot. The subtask label changes at the tick boundaries of the annotation (tick = 25 frames = 1.0 s). State and action follow the RoboDojo LeRobot v2.1 release unchanged.

Marked labels per task

Intended use

Train low-level policies that read the target position from the keyframe slot (3 live views + keyframe, per-frame subtask text) and evaluate them with a high-level planner that names positions as coordinates: at inference the planner's label is rewritten with the same rule and the marker is drawn on the current head frame before it reaches the policy.

Source

RoboDojo: https://github.com/RoboDojo-Benchmark/RoboDojo (long-horizon suite; dual ARX X5 in Isaac Sim). Labels: the TACOR offline annotator, task contexts of contexts/robodojo in the TACOR repository.