Myungkyu/real_workbench-taco-keyframe-gemini

real_workbench-taco-keyframe-gemini

Myungkyu/real_workbench-taco-gemini plus a keyframe stream: the same 320 episodes (4 x 80), 133,910 frames at 10 fps, the same reviewed per-frame subtask labels, the same 224x126 frames and delta_eef actions, with a third video feature observation.image.keyframe that carries the Gemini-annotated visual memory. Everything below the keyframe section is identical to the parent dataset.

The keyframe stream

observation.image.keyframe is the exterior view as retrieved from memory, with the same crop and size as observation.image.exterior_camera:

• —On a frame whose subtask carries a retrieval query, the slot shows the stored keyframe whose caption matches the query — the frame at which that memory was written, not the current frame.
• —On every other frame the slot shows the current exterior frame (the slot is a copy of the head view, exactly as in real_workbench-preset-gemini).

Where the memories come from, per task:

• —Object Identification — two memories per episode. The first frame of Close the near compartment of the left drawer. is stored with the caption Object inside the near compartment of the left drawer (the compartment is open and its part visible at that moment), and likewise the first frame of Close the far compartment … with Object inside the far compartment of the left drawer. The first pick-up subtask (Pick up the <part> from the center tray and place it into the right container.) retrieves the near memory, the second pick-up the far memory. Nine episodes whose reviewed labels do not contain the drawer steps (5, 9, 16, 30, 44, 55, 56, 63, 74) store no memory and retrieve nothing.
• —Layout Reconstruction — one memory per episode: frame 0 is stored as Initial object layout on the center tray, and every returning subtask (Pick up the <part> from the far board and place it into cell (c, r) of the center tray.) retrieves it.
• —Movement Reversal / Object Classification — no memory operations; the slot always shows the current exterior frame.

meta/keyframe_map.json lists, for every episode, the stored keyframes (frame index + caption) and each retrieval span (first frame, last frame, query, source frame), so the stream can be regenerated or checked without the video.

Layout Reconstruction names the destination of each returning object as a cell of a 4 x 4 grid on the center tray, cell (column, row) in the robot's frame — the robot stands to the right of the workbench facing left, so column 1 is its leftmost column and row 1 the row nearest it; cell (1, 1) is the near-left corner and (4, 4) the far-right one.

The workbench

Four fixtures, named the same way in every task: the center tray (the square plate in the middle), the left drawer (two compartments, near and far), the right container (the lidded box) and the far board (the stand at the back, left and right half). Four parts: the white holder, the metal ring, the black bracket and the black round base.

Frames and actions

The exterior view is cropped to the bottom-right 80 % x 80 % of the 1280x720 capture (the left fifth of the width and the top fifth of the height are dropped) and stored at 224x126; the wrist view is uncropped at 224x126; the keyframe stream is derived from the cropped exterior view. π0.5's resize_with_pad turns 224x126 into 224x224 with 49-px bars and no rescaling, so the stored frame is exactly the policy input. action is 7-D delta_eef (6-D cartesian velocity + gripper); observation.state is the 8-D joint vector (7 joints + gripper).

Source

Episodes: Myungkyu/object_retrieval, layout_reconstruction, movement_reversal, object_classification — DROID-style teleoperation captures, Meta Quest controller, cartesian-velocity control at 10 Hz, two ZED cameras, Apache-2.0. Labels: the reviewed TASCO annotation of real_workbench-taco-gemini. This dataset adds the keyframe stream only.