macrodata/WGO-Bench

WGO-Bench: What's Going On Benchmark

WGO-Bench is a small, manually annotated benchmark for evaluating how well vision-language models can turn robot and egocentric manipulation videos into timestamped subtask annotations.

Each row contains one video episode, a high-level task instruction, and gold subtask segments with start time, end time, and a concise action label. The benchmark is designed for two related tasks:

• —Boundary detection: predict where one meaningful manipulation subtask ends and the next begins.
• —Subtask labeling: given a fixed segment, describe the completed manipulation event.

Dataset Summary

Sources

Schema

DROID retains EEF position, rotation, and gripper state. Galaxea retains bilateral arm joints and grippers plus its available EEF- and gripper-derived channels. Actions are not included. HomER has no robot telemetry, so its rows contain null rather than synthetic zeros.

Quick Start

from datasets import load_dataset

dataset = load_dataset("macrodata/WGO-Bench", split="train")
example = dataset[0]

print(example["instruction"])
print(example["segments"])
print(example["perception_state"])

To use the parquet directly:

import pandas as pd

df = pd.read_parquet("hf://datasets/macrodata/WGO-Bench/data/annotations.parquet")

Rebuilding Robot States

scripts/perception_sources.lock.json pins both annotation maps and all 26 upstream robot-dataset revisions. Rebuild the enriched table from the original annotations Parquet:

python scripts/embed_perception_states.py \
  data/annotations.original.parquet \
  data/annotations.parquet

Requirements are Python 3.11+, huggingface_hub, and pyarrow. The exporter validates episode coverage, frame counts, channel widths, and timestamp order, then writes a SHA-256 provenance file next to the enriched Parquet. Hugging Face caches all state sources; pass --cache-dir to use a shared cache. The DROID state data comes from one roughly 241 MB upstream tar archive. The embedded state payload adds roughly 1.5 MB before Parquet-level recompression.

Annotation Policy

Segments are intended to describe completed manipulation events, not every small pose adjustment. A new segment should generally correspond to a visible state change such as picking up, placing, opening, closing, moving, pouring, wiping, cutting, or transferring an object.

The labels are short action descriptions grounded in what is visible in the clip. They are meant to be useful for downstream policy training, reward modeling, and evaluating automatic subtask annotation pipelines.

Related Resources

• —Blog post: Segmenting Robot Video into Actionable Subtasks
• —Refiner example: Subtask annotations
• —Code: macrodata-labs/refiner