InternRobotics/RoboInter-Data

1# RoboInter-Data: LeRobot v2.1 Format (Actions + Annotations + Videos)2 3The primary data format of [RoboInter-Data](https://huggingface.co/datasets/InternRobotics/RoboInter-Data). Contains robot **actions**, camera **observations**, and rich **intermediate representation annotations** in [LeRobot v2.1](https://github.com/huggingface/lerobot) format (parquet + MP4 videos), ready for policy training. Especially, we calculate the delta EEF (gripper) action of Droid (instead of the joint velocity or the origin cartesian action of the base).4 5| Sub-dataset | Source | Robot | Episodes | Frames | Tasks | Image Size | Raw Image Size |6|-------------|--------|-------|----------|--------|-------|------------|-------|7| `lerobot_droid_anno` | [DROID](https://droid-dataset.github.io/) | Franka + Robotiq | 152,986 | 46,259,014 | 43,026 | 320 x 180 | 640 x 360 |8| `lerobot_rh20t_anno` | [RH20T](https://rh20t.github.io/) | Multiple | 82,894 | 40,755,632 | 146 | 320 x 180 | 640 x 360 |9 10Both datasets share `fps=10`, `chunks_size=1000`, and the same annotation schema.11 12## Directory Layout13 14```15lerobot_droid_anno/  (or lerobot_rh20t_anno/)16├── meta/17│   ├── info.json              # Dataset metadata (fps, features, shapes, etc.)18│   ├── episodes.jsonl         # Per-episode info (one JSON per line)19│   ├── episodes_stats.jsonl   # Per-episode statistics20│   └── tasks.jsonl            # Task/instruction mapping21├── data/22│   └── chunk-{NNN}/           # Parquet data chunks (1,000 episodes per chunk)23│       ├── episode_000000.parquet24│       ├── episode_000001.parquet25│       └── ...26└── videos/27    └── chunk-{NNN}/28        ├── observation.images.primary/29        │   └── episode_{NNNNNN}.mp430        └── observation.images.wrist/31            └── episode_{NNNNNN}.mp432```33 34---35 36## Data Fields37 38### Core Fields (Shared by DROID & RH20T)39 40| Field | Shape | Type | Description |41|-------|-------|------|-------------|42| `action` | (7,) | float64 | Delta EEF action: [delta_x, delta_y, delta_z, delta_rx, delta_ry, delta_rz, gripper_command] |43| `state` | (7,) | float64 | EEF state: [x, y, z, rx, ry, rz, gripper_state] |44| `observation.images.primary` | (180, 320, 3) | video (H.264) | Primary camera RGB video |45| `observation.images.wrist` | (180, 320, 3) | video (H.264) | Wrist camera RGB video |46 47### Metadata Fields (Shared)48 49| Field | Type | Description |50|-------|------|-------------|51| `episode_name` | string | Episode unique identifier, e.g. `"3072_exterior_image_1_left"` |52| `camera_view` | string | Camera perspective, e.g. `"exterior_image_1_left"` |53| `task` | string | Task language description (via `task_index` -> `tasks.jsonl`) |54| `episode_index` | int64 | Episode index in dataset |55| `frame_index` | int64 | Frame index within episode |56| `timestamp` | float32 | Timestamp in seconds (`frame_index / fps`) |57| `index` | int64 | Global frame index across all episodes |58| `task_index` | int64 | Index into `tasks.jsonl` |59 60---61 62### Other Information Fields — DROID Only63 64`lerobot_droid_anno` contains the following additional fields from the original DROID dataset:65 66| Field | Shape | Type | Description |67|-------|-------|------|-------------|68| `other_information.language_instruction_2` | (1,) | string | Alternative language instruction (source 2) |69| `other_information.language_instruction_3` | (1,) | string | Alternative language instruction (source 3) |70| `other_information.action_delta_tcp_pose` | (7,) | float64 | Delta TCP pose action: [dx, dy, dz, drx, dry, drz, gripper] |71| `other_information.action_delta_wrist_pose` | (7,) | float64 | Delta wrist pose action: [dx, dy, dz, drx, dry, drz, gripper] |72| `other_information.action_tcp_pose` | (7,) | float64 | Absolute TCP pose: [x, y, z, rx, ry, rz, gripper] |73| `other_information.action_wrist_pose` | (7,) | float64 | Absolute wrist pose: [x, y, z, rx, ry, rz, gripper] |74| `other_information.action_gripper_velocity` | (1,) | float64 | Gripper velocity |75| `other_information.action_joint_position` | (7,) | float64 | Joint position action: [j1..j7] |76| `other_information.action_joint_velocity` | (7,) | float64 | Joint velocity action: [j1..j7] |77| `other_information.action_cartesian_velocity` | (6,) | float64 | Cartesian velocity: [vx, vy, vz, wx, wy, wz] |78| `other_information.observation_joint_position` | (7,) | float64 | Observed joint positions: [j1..j7] |79| `other_information.observation_gripper_position` | (1,) | float64 | Observed gripper position |80| `other_information.observation_gripper_open_state` | (1,) | float64 | Gripper open state |81| `other_information.observation_gripper_pose6d` | (6,) | float64 | Gripper 6D pose: [x, y, z, rx, ry, rz] |82| `other_information.observation_tcp_pose6d` | (6,) | float64 | TCP 6D pose: [x, y, z, rx, ry, rz] |83| `other_information.is_first` | (1,) | bool | First frame flag |84| `other_information.is_last` | (1,) | bool | Last frame flag |85| `other_information.is_terminal` | (1,) | bool | Terminal state flag |86 87### Other Information Fields — RH20T Only88 89`lerobot_rh20t_anno` contains the following additional fields from the original RH20T dataset:90 91| Field | Shape | Type | Description |92|-------|-------|------|-------------|93| `other_information.action_delta_tcp_pose` | (7,) | float64 | Delta TCP pose action: [dx, dy, dz, drx, dry, drz, gripper] |94| `other_information.action_tcp_pose` | (7,) | float64 | Absolute TCP pose: [x, y, z, rx, ry, rz, gripper] |95| `other_information.gripper_command` | (1,) | float64 | Gripper command |96| `other_information.observation_joint_position` | (14,) | float64 | Observed joint positions: [j1..j14] |97| `other_information.observation_gripper_open_state` | (1,) | float64 | Gripper open state |98| `other_information.observation_gripper_pose6d` | (6,) | float64 | Gripper 6D pose: [x, y, z, rx, ry, rz] |99| `other_information.tcp_camera` | (7,) | float64 | TCP in camera frame: [x, y, z, qx, qy, qz, qw] |100| `other_information.tcp_base` | (7,) | float64 | TCP in base frame: [x, y, z, qx, qy, qz, qw] |101| `other_information.gripper` | (1,) | string | Gripper metadata (JSON) |102| `other_information.is_first` | (1,) | bool | First frame flag |103| `other_information.is_last` | (1,) | bool | Last frame flag |104| `other_information.is_terminal` | (1,) | bool | Terminal state flag |105 106> **Key difference:** DROID has 7-DoF joint positions and richer action representations (wrist pose, joint/cartesian velocities). RH20T has 14-DoF joint positions, TCP transforms in camera/base frames, and gripper metadata JSON.107 108---109 110### Annotation Fields (Shared by DROID & RH20T)111 112All annotation fields are prefixed with `annotation.` and stored as JSON strings. Empty string `""` means no annotation is available for that frame.113 114| Field | Format | Description |115|-------|--------|-------------|116| `annotation.time_clip` | `[[start, end], ...]` | Subtask temporal segments (frame ranges) |117| `annotation.instruction_add` | string | Structured task language instruction |118| `annotation.substask` | string | Current subtask description |119| `annotation.primitive_skill` | string | Primitive skill label (pick, place, push, twist, etc.) |120| `annotation.segmentation` | string | Segmentation reference (path) |121| `annotation.object_box` | `[[x1, y1], [x2, y2]]` | Manipulated object bounding box |122| `annotation.placement_proposal` | `[[x1, y1], [x2, y2]]` | Target placement bounding box |123| `annotation.trace` | `[[x, y], ...]` | Future 10-frame gripper trajectory waypoints |124| `annotation.gripper_box` | `[[x1, y1], [x2, y2]]` | Gripper bounding box |125| `annotation.contact_frame` | int / -1 | Frame index when gripper contacts object (-1 = past contact) |126| `annotation.state_affordance` | `[x, y, z, rx, ry, rz]` | 6D EEF state at contact frame |127| `annotation.affordance_box` | `[[x1, y1], [x2, y2]]` | Gripper bounding box at contact frame |128| `annotation.contact_points` | `[x, y]` | Contact point in pixel coordinates |129| `annotation.origin_shape` | `[h, w]` | Original image resolution for coordinate reference |130 131#### Bounding Box Format132 133All bounding boxes use pixel coordinates with origin at top-left:134```json135[[x1, y1], [x2, y2]]   // [top-left, bottom-right]136```137 138#### Trace Format139 14010 future waypoints for gripper trajectory prediction:141```json142[[110, 66], [112, 68], [115, 70], [118, 72], [120, 75], [122, 78], [125, 80], [128, 82], [130, 85], [132, 88]]143```144 145---146 147### Q_Annotation Fields (Quality Indicators, Shared)148 149Each annotation has a corresponding quality indicator prefixed with `Q_annotation.`:150 151| Field | Values | Description |152|-------|--------|-------------|153| `Q_annotation.instruction_add` | `"Primary"` / `"Secondary"` / `""` | Instruction quality |154| `Q_annotation.substask` | `"Primary"` / `"Secondary"` / `""` | Subtask quality |155| `Q_annotation.primitive_skill` | `"Primary"` / `"Secondary"` / `""` | Primitive skill quality |156| `Q_annotation.segmentation` | `"Primary"` / `"Secondary"` / `""` | Segmentation quality |157| `Q_annotation.object_box` | `"Primary"` / `"Secondary"` / `""` | Object box quality |158| `Q_annotation.placement_proposal` | `"Primary"` / `"Secondary"` / `""` | Placement proposal quality |159| `Q_annotation.trace` | `"Primary"` / `"Secondary"` / `""` | Trace quality |160| `Q_annotation.gripper_box` | `"Primary"` / `"Secondary"` / `""` | Gripper box quality |161| `Q_annotation.contact_frame` | `"Primary"` / `"Secondary"` / `""` | Contact frame quality |162| `Q_annotation.state_affordance` | `"Primary"` / `"Secondary"` / `""` | State affordance quality |163| `Q_annotation.affordance_box` | `"Primary"` / `"Secondary"` / `""` | Affordance box quality |164| `Q_annotation.contact_points` | `"Primary"` / `"Secondary"` / `""` | Contact points quality |165 166- **Primary**: High-confidence annotation167- **Secondary**: Acceptable quality, may have minor errors168- **""** (empty): No annotation available169 170---171 172## Download & Extract173 174The `data/` and `videos/` directories are distributed as `.tar` archives (one per chunk) to reduce the number of files during transfer. After downloading, extract them in place:175 176```bash177cd Annotation_with_action_lerobotv21178 179for dataset in lerobot_droid_anno lerobot_rh20t_anno; do180  for subdir in data videos; do181    cd ${dataset}/${subdir}182    for f in *.tar; do tar xf "$f" && rm "$f"; done183    cd ../..184  done185done186```187 188After extraction, each `data/` will contain `chunk-000/`, `chunk-001/`, ... with `.parquet` files, and each `videos/` will contain `chunk-000/`, `chunk-001/`, ... with `.mp4` files. The `meta/` directories are ready to use without extraction.189 190 191## Quick Start192 193The dataloader code is at [RoboInterData/lerobot_dataloader](https://github.com/InternRobotics/RoboInter/tree/main/RoboInterData/lerobot_dataloader).194 195### Installation196 197```bash198pip install numpy torch pyarrow av opencv-python199```200 201### Basic Usage202 203```python204from lerobot_dataloader import create_dataloader205 206dataloader = create_dataloader(207    "path/to/lerobot_droid_anno",208    batch_size=32,209    action_horizon=16,210)211 212for batch in dataloader:213    images = batch["observation.images.primary"]  # (B, H, W, 3)214    actions = batch["action"]                     # (B, 16, 7)215    trace = batch["annotation.trace"]             # Parsed JSON lists216    skill = batch["annotation.primitive_skill"]   # List of strings217    break218```219 220### Multiple Datasets (DROID + RH20T)221 222```python223dataloader = create_dataloader(224    [225        "path/to/lerobot_droid_anno",226        "path/to/lerobot_rh20t_anno",227    ],228    batch_size=32,229    action_horizon=16,230)231 232for batch in dataloader:233    print(batch["dataset_name"])  # Source dataset identifier234    break235```236 237### Data Filtering238 239#### Frame Range Filtering240 241Remove idle frames at episode start/end using `range_nop.json`:242 243```python244dataloader = create_dataloader(245    "path/to/lerobot_droid_anno",246    range_nop_path="path/to/range_nop.json",247)248```249 250Format of `range_nop.json`:251```json252{253  "3072_exterior_image_1_left": [12, 217, 206]254}255```256`[start_frame, end_frame, valid_length]` — frames outside this range are idle/stationary.257 258#### Q_Annotation Filtering259 260Select episodes by annotation quality:261 262```python263from lerobot_dataloader import create_dataloader, QAnnotationFilter264 265# Only Primary quality266dataloader = create_dataloader(267    "path/to/lerobot_droid_anno",268    q_filters=[269        QAnnotationFilter("Q_annotation.instruction_add", ["Primary"]),270        QAnnotationFilter("Q_annotation.gripper_box", ["Primary"]),271    ]272)273 274# Any non-empty annotation275dataloader = create_dataloader(276    "path/to/lerobot_droid_anno",277    q_filters=[278        QAnnotationFilter("Q_annotation.trace", ["not_empty"])279    ]280)281```282 283#### Combined Filtering284 285```python286from lerobot_dataloader import FilterConfig, QAnnotationFilter287 288config = FilterConfig(289    range_nop_path="path/to/range_nop.json",290    q_filters=[291        QAnnotationFilter("Q_annotation.trace", ["Primary", "Secondary"]),292    ],293    q_filter_mode="all",  # "all" = AND, "any" = OR294)295 296dataloader = create_dataloader("path/to/lerobot_droid_anno", filter_config=config)297```298 299### Transforms300 301```python302from lerobot_dataloader import Compose, Normalize, ResizeImages, ToTensorImages, LeRobotDataset303from lerobot_dataloader.transforms import compute_stats304 305# Compute normalization stats306dataset = LeRobotDataset("path/to/lerobot_droid_anno", load_videos=False)307stats = compute_stats(dataset)308 309# Create transform pipeline310transform = Compose([311    ResizeImages(height=224, width=224),312    ToTensorImages(),   # (H,W,C) uint8 -> (C,H,W) float32313    Normalize(stats),314])315 316dataloader = create_dataloader("path/to/lerobot_droid_anno", transform=transform)317```318 319### Direct Dataset Access320 321```python322from lerobot_dataloader import LeRobotDataset323from lerobot_dataloader.transforms import ParseAnnotations324 325dataset = LeRobotDataset(326    "path/to/lerobot_droid_anno",327    transform=ParseAnnotations(),328)329 330print(f"Total frames: {len(dataset)}")331print(f"Total episodes: {dataset.num_episodes}")332print(f"FPS: {dataset.fps}")333 334sample = dataset[0]335print(f"Action: {sample['action']}")336print(f"Object box: {sample['annotation.object_box']}")337print(f"Skill: {sample['annotation.primitive_skill']}")338```339 340---341 342## Format Conversion343 344The LeRobot v2.1 format was converted from original data + LMDB annotations using:345 346- **DROID**: [convert_droid_to_lerobot_anno_fast.py](https://github.com/InternRobotics/RoboInter/blob/main/RoboInterData/convert_to_lerobot/convert_droid_to_lerobot_anno_fast.py)347- **RH20T**: [convert_rh20t_to_lerobot_anno_fast.py](https://github.com/InternRobotics/RoboInter/blob/main/RoboInterData/convert_to_lerobot/convert_rh20t_to_lerobot_anno_fast.py)348 349---350 351## Related Resources352 353| Resource | Link |354|----------|------|355| RoboInter-Data (parent dataset) | [HuggingFace](https://huggingface.co/datasets/InternRobotics/RoboInter-Data) |356| RoboInter Project | [GitHub](https://github.com/InternRobotics/RoboInter) |357| DataLoader Code | [lerobot_dataloader](https://github.com/InternRobotics/RoboInter/tree/main/RoboInterData/lerobot_dataloader) |358| Conversion Scripts | [convert_to_lerobot](https://github.com/InternRobotics/RoboInter/tree/main/RoboInterData/convert_to_lerobot) |359| Demo Visualizer | [RoboInterData-Demo](https://github.com/InternRobotics/RoboInter/tree/main/RoboInterData-Demo) |360| DROID Dataset | [droid-dataset.github.io](https://droid-dataset.github.io/) |361| RH20T Dataset | [rh20t.github.io](https://rh20t.github.io/) |362 363## License364 365Please refer to the original dataset licenses for [RoboInter](https://github.com/InternRobotics/RoboInter), [DROID](https://droid-dataset.github.io/), and [RH20T](https://rh20t.github.io/).366