Myungkyu/RoboDojo-taco-luna

RoboDojo-taco-luna

RoboDojo long-horizon episodes (8 tasks, 800 episodes, 25 fps) with dense high-level labels produced by the TACOR offline annotator: GPT-5.6 Luna (gpt-5.6-luna) reads the frames of each episode sampled every 25 frames as labelled images and labels every sampled frame under the task-specific context (taco) induced for that task, in its hybrid form: labels name the target object's image coordinates only where words cannot identify it (a random instance of a class that appears several times or at random places); unique objects and fixed places stay textual. The companion dataset RoboDojo-preset-luna carries the same episodes labelled with the subtask preset only.

Layout

<task>/                      LeRobot v2.1 dataset of one task (data/, meta/, videos/), episodes re-indexed 0..99
  videos/chunk-000/observation.image.head_camera/     head camera (RoboDojo cam_high, 640x480, 25 fps)
  videos/chunk-000/observation.image.left_wrist/      left wrist camera (cam_left_wrist)
  videos/chunk-000/observation.image.right_wrist/     right wrist camera (cam_right_wrist)
  videos/chunk-000/observation.image.keyframe/        the keyframe slot of the low-level policy; this benchmark uses no visual memory, so it is a copy of the head camera
  meta/dense_annotation_map.json                      per-episode subtask spans
  meta/robodojo_reindex.json                          RoboDojo episode index -> this dataset's index
retrieval_map.json           per task and episode (no retrievals in this benchmark)
annotations/<task>/ep%06d.jsonl   the raw per-tick labels (see below)

Per-frame features: observation.state (14: left arm 6 joints + gripper, right arm 6 joints + gripper), action (14), subtask (the label of the frame), global_task, subtask_end, episode_id, the three camera views and the keyframe slot. The subtask label changes at the tick boundaries of the annotation (tick = 25 frames = 1.0 s). State and action follow the RoboDojo LeRobot v2.1 release unchanged.

Tasks

Annotation rows

One JSON object per tick (decision_frame = 0, 25, 50, ...):

{"episode_index": 0, "decision_frame": 125, "subtask": "Place the left pen at (562, 407) in the pen holder.",
 "updated_memory": "...", "is_keyframe": false, "caption": "", "rag_query": ""}

subtask is a line of the task's catalogue with its placeholders filled. Where the label carries at (x, y), x and y are the target object's image position on a 0-1000 scale of the head-camera frame (x left to right, y top to bottom), read once at the first frame of the step; playtictactoe identifies pieces by an ordinal in their fixed row instead. `iskeyframe / caption / ragquery` are always false / empty in this benchmark. `updatedmemory` is the textual memory the planner carries. The prompts, the task contexts and the annotator are part of the TACOR code release.

Source

Episodes: RoboDojo (RoboDojo-Benchmark/RoboDojo, data/RoboDojo_lerobot_v21_video), long-horizon task suite, ARX X5 dual-arm simulation. RoboDojo is released under its own non-commercial research license; this dataset adds labels only and follows the same terms.