mwnuk/libero10-dense-object-mask-ecot

LIBERO-10 Object Masks + ECoT Reasoning Traces

Object-mask annotations and Embodied Chain-of-Thought (ECoT) reasoning traces for the 379 demonstrations of the LIBERO-10 (libero_10_image) benchmark.

Generated for the CoT-VLA project. Per-object segmentation masks were produced with interactive SAM2 point/box prompts and bidirectional video propagation; the CoT reasoning traces were hand-refined per task and re-timed to each episode's actuator (gripper + motion) signal.

Source data: derived from `lerobot/libero_10_image` (the LIBERO benchmark). This dataset contains only the derived annotations (masks, bounding boxes, reasoning traces) — not the original RGB frames. Use it alongside the source dataset for the images. Please credit LIBERO and the source dataset.

Contents

All tables join on episode_index (0–378). Frame-level tables also key on t.

Loading

from datasets import load_dataset

# per-frame reasoning traces
traces = load_dataset("mwnuk/libero10-dense-object-mask-ecot", "reasoning_traces", split="train")
print(traces[0]["subtask"], traces[0]["objects"])

# segment / phase structure
segs = load_dataset("mwnuk/libero10-dense-object-mask-ecot", "segments", split="train")

# bounding boxes (normalized xyxy)
boxes = load_dataset("mwnuk/libero10-dense-object-mask-ecot", "object_masks_bboxes", split="train")

Masks are side NPZ files (too large for parquet):

from huggingface_hub import hf_hub_download
import numpy as np
p = hf_hub_download("mwnuk/libero10-dense-object-mask-ecot", "object_masks/ep000000.npz", repo_type="dataset")
a = np.load(p, allow_pickle=True)
# a["masks"]:       [T, N, 256, 256] uint8 (0/1) — per-object binary masks
# a["object_names"][N], a["bboxes_t"]/[T,N,4], a["bboxes_t1"], a["objectness"]/[T,N]

Schema notes

• —Coordinate conventions. Masks and boxes are at the 256×256 annotation frame.
• —object_masks_bboxes.parquet: normalized xyxy in [0,1] (multiply by 256 for pixels).
• —reasoning_traces.parquet objects[].box: the same boxes in pixels [x1,y1,x2,y2] (0–256).
• —gripper: [u, v] pixel coordinates in the 256×256 frame.
• —Reasoning-trace fields (per frame): scene (scene description), plan (ordered high-level subtask list), subtask (current high-level subtask), gripper, action (discretized 7-DoF action in natural language), objects (visible objects + boxes).
• —`segments` decompose each episode into motion primitives (motion, e.g. lower, close gripper, move right) grouped under a high-level label (the subtask). Segment timing is derived from the episode's gripper/motion signal.
• —`phases` are the coarse grasp cycles (is_grasping toggling on gripper close/open).

Provenance & method

• —Masks: interactive SAM2 (hiera-large) point/box prompts → bidirectional video propagation.
• —Reasoning traces: one hand-refined template per task, propagated to all same-task episodes via piecewise-linear time-warping anchored on each episode's grasp transitions.
• —10 LIBERO-10 tasks, 379 episodes, all reviewed.

License & attribution

Annotations released under MIT. The underlying LIBERO imagery/trajectories are governed by the LIBERO benchmark's own license — consult the source dataset. If you use this data, please cite LIBERO and credit the source dataset.