InternRobotics/RoboInter-Data

1---2pretty_name: RoboInter-Data3task_categories:4- robotics5language:6- en7tags:8- Embodied-AI9- Robotic manipulation10- intermediate-representation11extra_gated_prompt: >-12  ### RoboInter-Data COMMUNITY LICENSE AGREEMENT13 14  All the data and code within this repo are under [CC BY-NC-SA15  4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/).16extra_gated_fields:17  First Name: text18  Last Name: text19  Email: text20  Country: country21  Affiliation: text22  Job title:23    type: select24    options:25    - Student26    - Research Graduate27    - AI researcher28    - AI developer/engineer29    - Other30  geo: ip_location31  By clicking Submit below I accept the terms of the license and acknowledge that the information I provide will be collected stored processed and shared in accordance with the RoboInter Privacy Policy: checkbox32extra_gated_description: >-33  The information you provide will be collected, stored, processed and shared in34  accordance with the RoboInter Privacy Policy.35extra_gated_button_content: Submit36---37 38# RoboInter-Data: Intermediate Representation Annotations for Robot Manipulation39 40Rich, dense, per-frame **intermediate representation annotations** for robot manipulation, built on top of [DROID](https://droid-dataset.github.io/) and [RH20T](https://rh20t.github.io/). Developed as part of the [RoboInter](https://github.com/InternRobotics/RoboInter) project. You can try our [**Online demo**](https://huggingface.co/spaces/wz7in/robointer-demo).41 42The annotations cover 230k episodes and include: subtasks,43primitive skills, segmentation, gripper/object bounding boxes, placement proposals, affordance boxes,44grasp poses, traces, contact points, etc. And each with a quality rating (Primary / Secondary).45 46## Dataset Structure47 48```49RoboInter-Data/50│51├── Annotation_with_action_lerobotv21/    # [Main] LeRobot v2.1 format (actions + annotations + videos)52│   ├── lerobot_droid_anno/               #   DROID: 152,986 episodes53│   └── lerobot_rh20t_anno/              #   RH20T:  82,894 episodes54│55├── Annotation_pure/                      # Annotation-only LMDB (no actions/videos)56│   └── annotations/                      #   35 GB, all 235,920 episodes57│58├── Annotation_raw/                       # Original unprocessed annotations59│   ├── droid_annotation.pkl              #   Raw DROID annotations (~20 GB)60│   ├── rh20t_annotation.pkl              #   Raw RH20T annotations (~11 GB)61│   └── segmentation_npz.zip.*            #   Segmentation masks (~50 GB, split archives)62│63├── Annotation_demo_app/                  # Small demo subset for online visualization64│   ├── demo_data/                        #   LMDB annotations for 20 sampled videos65│   └── videos/                           #   20 MP4 videos66│67├── Annotation_demo_larger/               # Larger demo subset for local visualization68│   ├── demo_annotations/                 #   LMDB annotations for 120 videos69│   └── videos/                           #   120 MP4 videos70│71├── All_Keys_of_Primary.json              # Episode names where all annotations are Primary quality72├── RoboInter_Data_Qsheet.json            # Per-episode quality ratings for each annotation type73├── RoboInter_Data_Qsheet_value_stats.json# Distribution statistics of quality ratings74├── RoboInter_Data_RawPath_Qmapping.json  # Mapping: original data source path -> episode splits & quality75├── range_nop.json                        # Non-idle frame ranges for all 230k episodes76├── range_nop_droid_all.json              # Non-idle frame ranges (DROID only)77├── range_nop_rh20t_all.json              # Non-idle frame ranges (RH20T only)78├── val_video.json                        # Validation set: 7,246 episode names79└── VideoID_2_SegmentationNPZ.json        # Episode video ID -> segmentation NPZ file path mapping80```81 82---83 84## 1. Annotation_with_action_lerobotv21 (Recommended)85 86The primary data format. Contains **actions + observations + annotations** in [LeRobot v2.1](https://github.com/huggingface/lerobot) format (parquet + MP4 videos), ready for policy training.87 88### Download & Extract89Dataset [link](https://huggingface.co/datasets/InternRobotics/RoboInter-Data/tree/main/Annotation_with_action_lerobotv21). Dataloader Code [link](https://github.com/InternRobotics/RoboInter/tree/main/RoboInterData).90The `data/` and `videos/` directories are distributed as `.tar` archives (one per chunk) to reduce the number of files during transfer. After downloading, extract them in place:91 92```bash93cd Annotation_with_action_lerobotv2194 95for dataset in lerobot_droid_anno lerobot_rh20t_anno; do96  for subdir in data videos; do97    cd ${dataset}/${subdir}98    for f in *.tar; do tar xf "$f" && rm "$f"; done99    cd ../..100  done101done102```103 104After extraction, each `data/` will contain `chunk-000/`, `chunk-001/`, ... with `.parquet` files, and each `videos/` will contain `chunk-000/`, `chunk-001/`, ... with `.mp4` files. The `meta/` directories are ready to use without extraction.105 106### Directory Layout107 108```109lerobot_droid_anno/  (or lerobot_rh20t_anno/)110├── meta/111│   ├── info.json              # Dataset metadata (fps=10, features, etc.)112│   ├── episodes.jsonl         # Episode information113│   └── tasks.jsonl            # Task/instruction mapping114├── data/115│   └── chunk-{NNN}/           # Parquet files (1,000 episodes per chunk)116│       └── episode_{NNNNNN}.parquet117└── videos/118    └── chunk-{NNN}/119        ├── observation.images.primary/120        │   └── episode_{NNNNNN}.mp4121        └── observation.images.wrist/122            └── episode_{NNNNNN}.mp4123```124 125### Data Fields126 127| Category | Field | Shape / Type | Description |128|----------|-------|-------------|-------------|129| **Core** | `action` | (7,) float64 | Delta EEF: [dx, dy, dz, drx, dry, drz, gripper] |130| | `state` | (7,) float64 | EEF state: [x, y, z, rx, ry, rz, gripper] |131| | `observation.images.primary` | (180, 320, 3) video | Primary camera RGB |132| | `observation.images.wrist` | (180, 320, 3) video | Wrist camera RGB |133| **Annotation** | `annotation.instruction_add` | string | Structured task language instruction |134| | `annotation.substask` | string | Current subtask description |135| | `annotation.primitive_skill` | string | Primitive skill label (pick, place, push, ...) |136| | `annotation.object_box` | JSON `[[x1,y1],[x2,y2]]` | Manipulated object bounding box |137| | `annotation.gripper_box` | JSON `[[x1,y1],[x2,y2]]` | Gripper bounding box |138| | `annotation.trace` | JSON `[[x,y], ...]` | Future 10-step gripper trajectory |139| | `annotation.contact_frame` | JSON int | Frame index when gripper contacts object |140| | `annotation.contact_points` | JSON `[x, y]` | Contact point pixel coordinates |141| | `annotation.affordance_box` | JSON `[[x1,y1],[x2,y2]]` | Gripper box at contact frame |142| | `annotation.state_affordance` | JSON `[x,y,z,rx,ry,rz]` | 6D EEF state at contact frame |143| | `annotation.placement_proposal` | JSON `[[x1,y1],[x2,y2]]` | Target placement bounding box |144| | `annotation.time_clip` | JSON `[[s,e], ...]` | Subtask temporal segments |145| **Quality** | `Q_annotation.*` | string | Quality rating: `"Primary"` / `"Secondary"` / `""` |146 147### Quick Start148The dataloader is located at our RoboInter [Codebase](https://github.com/InternRobotics/RoboInter/blob/main/RoboInterData/lerobot_dataloader).149 150```python151from lerobot_dataloader import create_dataloader152 153# Single dataset154dataloader = create_dataloader(155    "path/to/Annotation_with_action_lerobotv21/lerobot_droid_anno",156    batch_size=32,157    action_horizon=16,158)159 160for batch in dataloader:161    images = batch["observation.images.primary"]   # (B, H, W, 3)162    actions = batch["action"]                      # (B, 16, 7)163    trace = batch["annotation.trace"]              # JSON strings164    skill = batch["annotation.primitive_skill"]    # List[str]165    break166 167# Multiple datasets (DROID + RH20T)168dataloader = create_dataloader(169    [170        "path/to/lerobot_droid_anno",171        "path/to/lerobot_rh20t_anno",172    ],173    batch_size=32,174    action_horizon=16,175)176```177 178### Filtering by Quality & Frame Range179 180```python181from lerobot_dataloader import create_dataloader, QAnnotationFilter182 183dataloader = create_dataloader(184    "path/to/lerobot_droid_anno",185    batch_size=32,186    range_nop_path="path/to/range_nop.json",       # Remove idle frames187    q_filters=[188        QAnnotationFilter("Q_annotation.trace", ["Primary"]),189        QAnnotationFilter("Q_annotation.gripper_box", ["Primary", "Secondary"]),190    ],191)192```193 194For full dataloader documentation and transforms, see: [RoboInterData/lerobot_dataloader](https://github.com/InternRobotics/RoboInter/tree/main/RoboInterData/lerobot_dataloader).195 196### Format Conversion Scripts197 198The LeRobot v2.1 data was converted using:199 200- **DROID**: [convert_droid_to_lerobot_anno_fast.py](https://github.com/InternRobotics/RoboInter/blob/main/RoboInterData/convert_to_lerobot/convert_droid_to_lerobot_anno_fast.py)201- **RH20T**: [convert_rh20t_to_lerobot_anno_fast.py](https://github.com/InternRobotics/RoboInter/blob/main/RoboInterData/convert_to_lerobot/convert_rh20t_to_lerobot_anno_fast.py)202 203---204 205## 2. Annotation_pure (Annotation-Only LMDB)206 207Contains **only the intermediate representation annotations** (no action data, no videos) stored as a single LMDB database. Useful for lightweight access to annotations or as input for the LeRobot conversion pipeline. The format conversion scripts and corresponding lightweight dataloader functions are provided in [lmdb_tool](https://github.com/InternRobotics/RoboInter/blob/main/RoboInterData/lmdb_tool). You can downloade high-resolution208videos by following [Droid hr_video_reader](https://github.com/InternRobotics/RoboInter/blob/main/RoboInterData/hr_video_reader) and [RH20T API](https://github.com/rh20t/rh20t_api).209 210### Data Format211 212Each LMDB key is an episode name (e.g., `"3072_exterior_image_1_left"`). The value is a dict mapping frame indices to per-frame annotation dicts:213 214```python215{216    0: {  # frame_id217        "time_clip": [[0, 132], [132, 197], [198, 224]],   # subtask segments218        "instruction_add": "pick up the red cup",           # language instruction219        "substask": "reach for the cup",                    # current subtask220        "primitive_skill": "reach",                         # skill label221        "segmentation": None,                               # (stored separately in Annotation_raw)222        "object_box": [[45, 30], [120, 95]],                # manipulated object bbox223        "placement_proposal": [[150, 80], [220, 140]],      # target placement bbox224        "trace": [[x, y], ...],                             # next 10 gripper waypoints225        "gripper_box": [[60, 50], [100, 80]],               # gripper bbox226        "contact_frame": 101,                               # contact event frame (−1 if past contact)227        "state_affordance": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6],# 6D EEF state at contact228        "affordance_box": [[62, 48], [98, 82]],             # gripper bbox at contact frame229        "contact_points": [[75, 65], [85, 65]],             # contact pixel coordinates230        ...231    },232    1: { ... },233    ...234}235```236 237### Reading LMDB238 239```python240import lmdb241import pickle242 243lmdb_path = "Annotation_pure/annotations"244env = lmdb.open(lmdb_path, readonly=True, lock=False, readahead=False)245 246with env.begin() as txn:247    # List all episode keys248    cursor = txn.cursor()249    for key, value in cursor:250        episode_name = key.decode("utf-8")251        episode_data = pickle.loads(value)252 253        # Access frame 0254        frame_0 = episode_data[0]255        print(f"{episode_name}: {frame_0['instruction_add']}")256        print(f"  object_box: {frame_0['object_box']}")257        print(f"  trace: {frame_0['trace'][:3]}...")  # first 3 waypoints258        break259 260env.close()261```262 263### CLI Inspection Tool264 265```bash266cd RoboInter/RoboInterData/lmdb_tool267 268# Basic info269python read_lmdb.py --lmdb_path Annotation_pure/annotations --action info270 271# View a specific episode272python read_lmdb.py --lmdb_path Annotation_pure/annotations --action item --key "3072_exterior_image_1_left"273 274# Field coverage statistics275python read_lmdb.py --lmdb_path Annotation_pure/annotations --action stats --key "3072_exterior_image_1_left"276 277# Multi-episode summary278python read_lmdb.py --lmdb_path Annotation_pure/annotations --action summary --limit 100279```280 281---282 283## 3. Annotation_raw (Original Annotations)284 285The original, unprocessed annotation files before conversion to LMDB format. These files are large and slow to load.286 287| File | Size | Description |288|------|------|-------------|289| `droid_annotation.pkl` | ~20 GB | Raw DROID intermediate representation annotations |290| `rh20t_annotation.pkl` | ~11 GB | Raw RH20T intermediate representation annotations |291| `segmentation_npz.zip.*` | ~50 GB | Object segmentation masks (split archives) |292 293### Reading Raw PKL294```bash295cd /RoboInter-Data/Annotation_raw296cat segmentation_npz.zip.* > segmentation_npz.zip297unzip segmentation_npz.zip298```299 300```python301import pickle302 303with open("Annotation_raw/droid_annotation.pkl", "rb") as f:304    droid_data = pickle.load(f)  # Warning: ~20 GB, takes several minutes305 306# droid_data[episode_key] contains raw intermediate representation data307# including: all_language, all_gripper_box, all_grounding_box, all_contact_point, all_traj, etc.308```309 310> To convert raw PKL to the LMDB format used in `Annotation_pure`, see the conversion script in the [RoboInter repository](https://github.com/InternRobotics/RoboInter).311 312---313 314## 4. Demo Subsets (Annotation_demo_app & Annotation_demo_larger)315 316Pre-packaged subsets for quick visualization using the [RoboInterData-Demo](https://github.com/InternRobotics/RoboInter/tree/main/RoboInterData-Demo) Gradio app. Both subsets share the same LMDB annotation format + MP4 video structure.317 318| Subset | Videos | Size | Use Case |319|--------|--------|------|----------|320| `Annotation_demo_app` | 20 | ~929 MB | HuggingFace Spaces [online demo](https://huggingface.co/spaces/wz7in/robointer-demo) |321| `Annotation_demo_larger` | 120 | ~12 GB | Local visualization with more examples |322 323### Running the Visualizer324 325```bash326git clone https://github.com/InternRobotics/RoboInter.git327cd RoboInter/RoboInterData-Demo328 329# Option A: Use the small demo subset (for Spaces)330ln -s /path/to/Annotation_demo_app/demo_data ./demo_data331ln -s /path/to/Annotation_demo_app/videos ./videos332 333# Option B: Use the larger demo subset (for local)334ln -s /path/to/Annotation_demo_larger/demo_annotations ./demo_data335ln -s /path/to/Annotation_demo_larger/videos ./videos336 337pip install -r requirements.txt338python app.py339# Open http://localhost:7860340```341 342The visualizer supports all annotation types: object segmentation masks, gripper/object/affordance bounding boxes, trajectory traces, contact points, grasp poses, and language annotations (instructions, subtasks, primitive skills).343 344---345 346## 5. Metadata JSON Files347 348### Quality & Filtering349 350| File | Description |351|------|-------------|352| `All_Keys_of_Primary.json` | List of 65,515 episode names where **all** annotation types are rated Primary quality. |353| `RoboInter_Data_Qsheet.json` | Per-episode quality ratings for every annotation type. Each entry contains `Q_instruction_add`, `Q_substask`, `Q_trace`, etc. with values `"Primary"`, `"Secondary"`, or `null`. |354| `RoboInter_Data_Qsheet_value_stats.json` | Distribution of quality ratings across all episodes. |355| `RoboInter_Data_RawPath_Qmapping.json` | Mapping from original data source paths to episode splits and their quality ratings. |356 357### Frame Ranges (Idle Frame Removal)358 359| File | Description |360|------|-------------|361| `range_nop.json` | Non-idle frame ranges for all 235,920 episodes (DROID + RH20T). |362| `range_nop_droid_all.json` | Non-idle frame ranges for DROID episodes only. |363| `range_nop_rh20t_all.json` | Non-idle frame ranges for RH20T episodes only. |364 365Format: `{ "episode_name": [start_frame, end_frame, valid_length] }`366 367```python368import json369 370with open("range_nop.json") as f:371    range_nop = json.load(f)372 373# Example: "3072_exterior_image_1_left": [12, 217, 206]374# Means: valid action frames are 12~217, total 206 valid frames375# (frames 0~11 and 218+ are idle/stationary)376```377 378### Other379 380| File | Description |381|------|-------------|382| `val_video.json` | List of 7,246 episode names reserved for the validation set. |383| `VideoID_2_SegmentationNPZ.json` | Mapping from episode video ID to the corresponding segmentation NPZ file path in `Annotation_raw/segmentation_npz`. `null` if no segmentation is available. |384 385---386 387## Related Resources388 389| Resource | Link |390|----------|------|391| Project | [RoboInter](https://github.com/InternRobotics/RoboInter) |392| VQA Dataset | [RoboInter-VQA](https://huggingface.co/datasets/InternRobotics/RoboInter-VQA) |393| VLM Checkpoints | [RoboInter-VLM](https://huggingface.co/InternRobotics/RoboInter-VLM) |394| LMDB Tool | [RoboInterData/lmdb_tool](https://github.com/InternRobotics/RoboInter/tree/main/RoboInterData/lmdb_tool) |395| High-Resolution Video Reader | [RoboInterData/hr_video_reader](https://github.com/InternRobotics/RoboInter/tree/main/RoboInterData/hr_video_reader) |396| LeRobot DataLoader | [RoboInterData/lerobot_dataloader](https://github.com/InternRobotics/RoboInter/tree/main/RoboInterData/lerobot_dataloader) |397| LeRobot Conversion | [RoboInterData/convert_to_lerobot](https://github.com/InternRobotics/RoboInter/tree/main/RoboInterData/convert_to_lerobot) |398| Demo Visualizer | [RoboInterData-Demo](https://github.com/InternRobotics/RoboInter/tree/main/RoboInterData-Demo) |399| Online Demo | [HuggingFace Space](https://huggingface.co/spaces/wz7in/robointer-demo) |400| Raw DROID Dataset | [droid-dataset.github.io](https://droid-dataset.github.io/) |401| Raw RH20T Dataset | [rh20t.github.io](https://rh20t.github.io/) |402 403## Citation404 405If you find RoboInter useful in your research, please consider citing:406 407```bibtex408@article{li2026robointer,409  title={RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulation},410  author={Li, Hao and Wang, Ziqin and Ding, Zi-han and Yang, Shuai and Chen, Yilun and Tian, Yang and Hu, Xiaolin and Wang, Tai and Lin, Dahua and Zhao, Feng and others},411  journal={arXiv preprint arXiv:2602.09973},412  year={2026}413}414```415 416## License417 418Please refer to the original dataset licenses for [RoboInter](https://github.com/InternRobotics/RoboInter), [DROID](https://droid-dataset.github.io/), and [RH20T](https://rh20t.github.io/).419