agibot-world/AgiBotWorld2026

<div align="center">

AgiBot World 2026

Real-World Embodied Intelligence Dataset

  

<br/>

<img src="./assets/teaser.gif" alt="AgiBot World 2026 dataset preview (animated)" style="width: 100%; max-width: 100%; height: auto; display: block; margin: 0 auto;" />

<br/>

</div>

Overview

As robotics research advances into real-world scenarios, the demand for authentic, high-quality data has become increasingly urgent. Following AGIBOT WORLD's "ImageNet moment," we now release the AGIBOT WORLD 2026 dataset. Built upon massive real-world scenes, it systematically spans pivotal research directions in embodied intelligence, designed to power the next generation of embodied agents.

The AGIBOT WORLD 2026 dataset is collected from 100% real-world environments, covering commercial spaces, home, and other general-purpose scenarios. Collected on the AGIBOT G2 robot platform through a free-form collection mode, the dataset provides developers with structured, accurately annotated, high-quality data. Digital twin technology is leveraged to construct a 1:1 scale scenario in the simulation environment for data collection, with the simulation data concurrently open-sourced in the GenieSim project.

We invite researchers worldwide to leverage AGIBOT WORLD 2026 to drive robotic intelligence from the lab into the real world, empowering every industry and tangibly boosting production and service efficiency.

Get Started

Download the Dataset

To download the full dataset, you can use the following code. If you encounter any issues, please refer to the official Hugging Face documentation.

# Make sure you have git-lfs installed (https://git-lfs.com)
git lfs install

# When prompted for a password, use an access token with write permissions.
# Generate one from your settings: https://huggingface.co/settings/tokens
git clone https://huggingface.co/datasets/agibot-world/AgiBotWorld2026

# If you want to clone without large files - just their pointers
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/datasets/agibot-world/AgiBotWorld2026

If you only want to download a specific task, such as task_3777, you can use the following code.

# Make sure you have git-lfs installed (https://git-lfs.com)
git lfs install

# Initialize an empty Git repository
git init AgiBotWorld2026
cd AgiBotWorld2026

# Set the remote repository
git remote add origin https://huggingface.co/datasets/agibot-world/AgiBotWorld2026

# Enable sparse-checkout
git sparse-checkout init

# Specify the folders and files
git sparse-checkout set 20260315/tar/task_3777 20260315/task_3777

# Pull the data
git pull origin main

To facilitate the inspection of the dataset's internal structure and examples, we also provide a sample dataset (~7 GB). Please refer to task3777/380098_380609.tar.gz.

Dataset Structure

LeRobot version reference: v2.1

Overview

A LeRobot dataset is organized into three main parts:

Directory Layout

dataset_root/
├── meta/
│   ├── episodes.jsonl
│   ├── info.json
│   ├── episodes_stats.jsonl
│   ├── annotations.json
│   └── tasks.jsonl
├── data/
│   └── chunk-000/
│       ├── episode_000000.parquet
│       ├── episode_000001.parquet
│       └── ...
└── videos/
    └── chunk-000/
        ├── observation.images.top_head/
        │   ├── episode_000000.mp4
        │   └── ...
        ├── observation.images.hand_left/
        │   ├── episode_000000.mp4
        │   └── ...
        └── observation.images.hand_right/
            ├── episode_000000.mp4
            └── ...

Core Storage Rules

• —Each Parquet file stores one complete episode.
• —Each video file stores one camera stream for one episode.
• —The dataset schema is primarily defined in meta/info.json.
• —File paths are generated from the templates in info.json:
• —data_path — path template for Parquet episode files
• —video_path — path template for MP4 video files

Metadata Files

episodes.jsonl

{"episode_index": 0, "tasks": ["G2"], "length": 1242}

info.json

info.json is the most important metadata file. It contains:

<details> <summary><b>Click to expand full schema</b></summary>

• —Dataset summary: robot_type, total_episodes, total_frames, fps, splits
• —File templates: data_path, video_path
• —Schema definition: features
• —Optional custom metadata: instruction_segments, key_frame, high_level_instruction, take_over, h5_path, camera_parameters

</details>

Features Schema

The features object defines how data is stored in Parquet files and how video streams are represented.

Main Feature Groups

State and Action Vectors

For observation.state and action, the actual values are stored in Parquet files, while the schema is described by field_descriptions.

Each entry in field_descriptions typically includes:

• —description — human-readable description of the field
• —dimensions — number of dimensions in this sub-field
• —dims — optional names for each dimension
• —indices — index positions inside the flattened Parquet vector

<details> <summary><b>Example keys</b></summary>

• —state/left_effector/position
• —state/right_effector/position
• —state/joint/position
• —state/joint/velocity
• —action/joint/position
• —action/robot/velocity

</details>

Camera Feature Keys

Camera streams are represented as observation.images.* keys. Each one describes video type, frame rate, codec, pixel format, and frame shape.

<details> <summary><b>Common camera keys</b></summary>

• —observation.images.top_head
• —observation.images.hand_left
• —observation.images.hand_right
• —observation.images.head_depth
• —observation.images.head_left_fisheye
• —observation.images.head_right_fisheye
• —observation.images.head_back_fisheye_color
• —observation.images.head_stereo_left_color
• —observation.images.head_stereo_right_color

</details>

Annotation Layers

The dataset follows the LeRobot v2.1 directory structure. meta/info.json adds three annotation layers beyond the standard LeRobot spec, stored under the key_frame and instruction_segments keys, indexed by episode index (as a string).

Episode ── high-level label (tasks.jsonl)
│
└── Task Frame segments ── subtask instruction + [start, end)
    │
    ├── 2D Bounding Box ── object label + bbox + camera
    │
    └── Instruction Segments ── skill + step instruction + [start, end)

Layer 1 — Task Frame (Subtask Instructions)

key_frame[ep_idx]["dual"] entries with "frame_type_name": "Task Frame" define subtask-level segments. Each entry specifies a time interval [start, end) in frame indices and a natural language instruction in frame_detail.comment.

{
  "track": "subtask",
  "frame_type_name": "Task Frame",
  "start": 35,
  "end": 2516,
  "comment": "",
  "frame_detail": {
    "comment": "Place the red-capped drinks and the white triple-pack yogurts from the shopping cart into the fifth shelf of the refrigerated cabinet.",
    "is_result_succeed": true
  }
}

A single long-horizon episode may contain multiple Task Frame entries — each covering a distinct subtask the robot must complete. This is the annotation used by split_episode.py to produce single-instruction episodes.

Layer 2 — 2D Bounding Box (Object Annotations)

key_frame[ep_idx]["dual"] entries with "frame_type_name": "2D Bounding Box" label key objects that the robot interacts with. Each entry includes the object category, the arm track it belongs to, the frame interval during which the object is relevant, and the bounding box in normalized coordinates.

{
  "track": "Right arm",
  "frame_type_name": "2D Bounding Box",
  "start": 493,
  "end": 743,
  "comment": "",
  "frame_detail": {
    "box": { "h": 0.239, "w": 0.083, "x": 0.490, "y": 0.574 },
    "camera": "head_color",
    "comment": "Yogurt, white",
    "type": "box"
  }
}

These annotations can be used to train object-conditioned policies, ground language instructions to visual regions, or study grasp selection.

Layer 3 — Instruction Segments (Step-Level Labels)

instruction_segments[ep_idx] is a list of fine-grained step-level segments, each covering a primitive skill (e.g., "Pick", "Bend waist forward") with a natural language instruction and frame boundaries.

{
  "track": "default",
  "skill": "Pick",
  "instruction": "The left arm picks up the red-capped drink from the shopping cart.",
  "instruction_augmentation": {},
  "start_frame_index": 284,
  "success_frame_index": 493,
  "end_frame_index": 493
}

This layer enables research into skill-level imitation learning, primitive discovery, and multi-granularity language conditioning.

Optional Annotation Fields in info.json

<details> <summary><b>instruction_segments</b></summary>

Per-episode step annotations. Each item may contain:

• —track — annotation track name
• —instruction — step description
• —start_frame_index — start frame of the segment
• —end_frame_index — end frame of the segment

</details>

<details> <summary><b>key_frame</b></summary>

Per-episode key frame annotations. Two common categories are:

• —`single` — single-frame annotations
• —`dual` — range annotations

Typical fields: track, frame_type_name, start, end, frame_detail.comment, frame_detail.error_cause, frame_detail.restorable, frame_detail.extra

Common built-in frame type names: Error Frame, Success Frame, Intervention Frame, Single Frame, 2D Bounding Box, Task Frame

</details>

Episode Data and Video Data

Practical Reading Order

1. 1.Read meta/info.json to get the schema and path templates.
2. 2.Read meta/episodes.jsonl to learn episode-level metadata.
3. 3.Read the target field_descriptions entries in features and collect the required indices.
4. 4.Load episode Parquet files from data/ and extract required dimensions.
5. 5.Load matching camera videos from videos/.

Installation

The following link from the Hugging Face team provides instructions for installing LeRobot, which requires Python 3.10+ and PyTorch 2.2+.

LeRobot Installation Guide

Two Usage Modes

Mode A — Original Format (Hierarchical Research)

Use the dataset as-is. Each episode covers a complete long-horizon restocking task. Access the multi-level annotations from meta/info.json to train hierarchical policies, task-conditioned planners, or multi-granularity instruction-following models.

import json

with open("meta/info.json") as f:
    info = json.load(f)

# Iterate subtask segments for episode 0
for entry in info["key_frame"]["0"]["dual"]:
    if entry["frame_type_name"] == "Task Frame":
        print(entry["start"], entry["end"], entry["frame_detail"]["comment"])

# Iterate step-level segments for episode 0
for step in info["instruction_segments"]["0"]:
    print(step["skill"], step["instruction"], step["start_frame_index"], step["end_frame_index"])

Mode B — Split Format (Standard LeRobot, Plug-and-Play)

A companion script split_episode.py is provided to convert the dataset into standard single-episode single-instruction LeRobot format, compatible with existing training pipelines out of the box.

Run split_episode.py to split long-horizon episodes into single-instruction episodes compatible with standard LeRobot training scripts (e.g., lerobot/scripts/train.py). The extended annotation fields are stripped from info.json in the output.

Tool: split_episodes_tool.zip

Prerequisites: Python 3.8+ &nbsp;|&nbsp; FFmpeg in PATH

pip install pyarrow numpy

Usage

python split_episode.py <input_dataset_path> [--output_path <output>] [--num_workers <N>]

Example

python split_episode.py ./my_dataset --output_path ./my_dataset_split --num_workers 8

<details> <summary><b>What happens (demo example)</b></summary>

Each resulting episode has exactly one instruction:

episode_000001 → "Put the pink bottled yogurt from the third layer of the chilled cabinet into the shopping cart."
episode_000002 → "Put the pink bottled yogurt from the third shelf of the chilled cabinet into the shopping cart."
episode_000003 → "Put the red bottled yogurt from the third shelf of the refrigerated case into the shopping cart."
episode_000004 → "Put the blue bottled yogurt from the third layer of the chilled display case into the shopping cart."

</details>

The script:

• —Copies the full dataset to the output path (original is never modified)
• —Slices parquet files by frame range; resets frame_index, index, timestamp, task_index
• —Extracts video segments via FFmpeg (preserves codec and pixel format)
• —Rewrites meta/info.json, episodes.jsonl, tasks.jsonl, episodes_stats.jsonl

License and Citation

All the data and code within this repo are licensed under **CC BY-NC-SA 4.0**. Please consider citing our project if it contributes to your research.

@misc{agibotworld2026,
    title        = {AgiBot World 2026},
    author       = {AgiBot World Team},
    howpublished = {\url{https://huggingface.co/datasets/agibot-world/AgiBotWorld2026}},
    year         = {2026}
}