Myungkyu/RMBench-taco-wodemo-luna

RMBench-taco-wodemo-luna

RMBench training episodes (9 tasks, 450 episodes, 30 fps) with dense high-level labels produced by the TACOR offline annotator: GPT-5.6 Luna (gpt-5.6-luna) reads the frames of each episode sampled every 25 frames as labelled images and labels every sampled frame under a task-specific context (taco) for that task. Each tick carries the current subtask, the running textual memory and the visual-memory operations (keyframe store / retrieval) that the online high-level planner reproduces at inference time.

TASCO-w/o-demo ablation. This dataset differs from RMBench-taco-luna only in how the task-specific context (the TASCO document) behind the annotation was written: Claude Opus 5 composed it from the task instruction and the fixed subtask preset ONLY, without seeing any reference demonstration. The subtask catalogue is the same preset label set, so the two datasets share their label vocabulary; what changes is the context document the annotator was given. The companion dataset RMBench-taco-wodemo-gemini carries the same episodes labelled by Gemini 3.7 Flash under the same context.

Layout

<task>/                      LeRobot v2.1 dataset of one task (data/, meta/, videos/)
  videos/chunk-000/observation.image.head_camera/     head camera
  videos/chunk-000/observation.image.left_wrist/      left wrist camera
  videos/chunk-000/observation.image.right_wrist/     right wrist camera
  videos/chunk-000/observation.image.keyframe/        the retrieved keyframe shown to the low-level policy at each frame
  meta/dense_annotation_map.json                      per-episode stored / retrieved frames and subtask spans
retrieval_map.json           per task and episode: keyframe frame + frames at which it is retrieved
annotations/<task>/ep%06d.jsonl   the raw per-tick labels (see below)

Per-frame features: observation.state, action, subtask (the label of the frame), subtask_end, the three camera views and the keyframe slot. The subtask label changes at the tick boundaries of the annotation (tick = 25 frames).

Tasks

Annotation rows

One JSON object per tick (decision_frame = 0, 25, 50, ...):

{"episode_index": 1, "decision_frame": 150, "subtask": "Place the red T-block at the right, tilted to the left.",
 "updated_memory": "...", "is_keyframe": false, "caption": "", "rag_query": "both T-block poses before the swap"}

subtask is a line of the task's subtask catalogue with its placeholders filled; updated_memory is the textual memory the planner carries; is_keyframe / caption store the current frame in the visual-memory bank under that caption; rag_query retrieves the stored frame whose caption matches. Tasks with counters (press_button) add a counts object. The prompts, the task-specific contexts and the annotator are part of the TACOR code release.