RoboDyna/robodyna-benchmark-v2

RoboDyna Benchmark

Expert demonstrations for RoboDyna, a dual-arm manipulation benchmark built around dynamic scenes — moving targets, rolling and falling objects, closing time windows, conveyor belts, and distractors — rather than static pick-and-place. Every episode is a scripted-expert rollout that succeeded; failures are not published.

Built on RoboTwin 2.0 / DOMINO with SAPIEN 3.0.3 and a dual-UR5 + WSG gripper embodiment (ur5-wsg).

At a glance

Layout

The two formats hold identical content and are indexed identically: hdf5/<task>/<op>/episodeN.hdf5 corresponds to LeRobot local episode index N.

hdf5/<task>/<op>/episode0.hdf5 … episode49.hdf5
lerobot/<task>/<op>/
├── data/task-XXXX/episode_XXXXXXXX.parquet
├── videos/task-XXXX/observation.images.{head,left_wrist,right_wrist}/episode_XXXXXXXX.mp4
├── annotations/task-XXXX/episode_XXXXXXXX.json
└── meta/{info.json,episodes.jsonl,episodes_stats.jsonl,tasks.jsonl}

<op> is one of base, opt1, opt2 (see Conditions).

Loading

The repository is large, so pull one combination at a time. Each lerobot/<task>/<op> directory is a self-contained LeRobot v2.1 dataset with its own meta/.

from huggingface_hub import snapshot_download

root = snapshot_download(
    "RoboDyna/robodyna-benchmark-v2", repo_type="dataset",
    allow_patterns="lerobot/hit_target/opt1/**",
) + "/lerobot/hit_target/opt1"

import glob, pyarrow.parquet as pq

table = pq.read_table(sorted(glob.glob(f"{root}/data/task-*/*.parquet"))[0])
print(table.num_rows, table.column_names)
# 95 ['observation.state', 'observation.endpose', 'observation.task_state',
#     'action', 'timestamp', 'frame_index', 'episode_index', 'index', 'task_index']

Images are not in the parquet — they are the MP4 files under videos/, one per camera per episode, which is how LeRobot v2.1 stores them. If you have lerobot installed, point LeRobotDataset at root and it will decode them for you.

The HDF5 copy is standalone and needs nothing but h5py:

import h5py

with h5py.File("hdf5/hit_target/opt1/episode0.hdf5") as f:
    actions = f["joint_action/vector"][:]           # (T, 14)
    rgb0 = f["observation/head_camera/rgb"][0]      # encoded image bytes
    hit = f["hit_target/center_hit"][:]             # per-task ground truth

RGB in the HDF5 files is stored as encoded image bytes (variable-length), not raw arrays — decode with cv2.imdecode / PIL.

What an episode contains

LeRobot features

observation.task_state is flattened and padded to a fixed width, so the column names live in the per-episode annotation JSON under dynamic_state.schema. In the HDF5 copy the same values are stored unpadded and self-describing, in a group named after the task:

hit_target/target_center_world   (T, 3)     hit_target/blocker_speed     (T,)
hit_target/dart_tip_world        (T, 3)     hit_target/center_hit        (T,) bool
hit_target/hit_ring_index        (T,) int   hit_target/hit_score         (T,)

Every task exposes its own such group — the moving element's world position and velocity, the per-frame success predicate, and whatever the scene randomised. This is what makes the dataset usable for studying timing and not just trajectories.

Each annotation JSON also carries the natural-language instruction and a primitive_annotation segment list.

Cameras

All three cameras are Intel D435 models at 320×240. The head camera is at the same world pose for every task and every episode — position (0.00, −0.50, 2.00), looking down and forward, 37° vertical FOV (fx = fy = 358.6, cx = 160, cy = 120) — so head-view policies transfer across the whole suite without a per-task calibration step. Per-frame intrinsic_cv, extrinsic_cv and cam2world_gl are recorded in the HDF5 files.

Depth, point clouds and segmentation were not collected. (An empty pointcloud dataset is present in the HDF5 files for schema compatibility — ignore it.)

Conditions

23 dynamic tasks ship three conditions each (base, opt1, opt2); each condition toggles one independent difficulty axis on top of base, so the pair isolates two factors rather than stacking them.

12 household tasks ship base only: boil_milk, catch_cup, catch_mouse_object_drop, clean_table, cook_food, cook_food_timer, fill_coffee_jar, make_soup, measure_ingredient, pour_beer, stop_ball, trap_bug.

23 × 3 + 12 = 81 combinations.

Collection protocol

Each combination was collected by a scripted expert policy over randomly seeded scenes, in a two-pass process: pass 1 plans and banks trajectories, pass 2 replays each banked trajectory with rendering on. The collector retries new seeds until 50 rollouts satisfy the task's own check_success predicate, so every combination is exactly 50 successful episodes and success rate is not encoded in the episode count.

Quality checks run on the published corpus:

• —No duplicate episodes. Every episode was signed by md5(joint_action) + md5(first head frame) + md5(last head frame); all 4,050 signatures are distinct. An earlier signature over joint_action alone over-reported, because several tasks randomise things the scripted arm path never reacts to (gummy colours, catch_cuboid's opaque surface) and therefore share an action stream across genuinely different scenes.
• —Condition overrides verified to take effect, by comparing mean episode length across conditions (e.g. hit_target 124 / 145 / 279 frames for base / opt1 / opt2).
• —Per-combination counts verified against both formats: 50 HDF5 files, 50 parquet files, and meta/info.json → total_episodes == 50.

Known limitations

• —Only successful rollouts are included — there is no failure data for reward learning or failure detection.
• —seed.txt files are not published, and seeds are not portable anyway: pass 2 replays a locally cached trajectory, so a seed reproduces a scene only on the machine that planned it.
• —The demonstrations are scripted, not teleoperated. Motions are efficient but not human-like.
• —observation.task_state is zero-padded to 32 dims; read dynamic_state.schema from the annotation before slicing it.

License

Apache-2.0, following the RoboDyna codebase and its RoboTwin 2.0 upstream. Individual 3D assets carry their own notices in the source repository.