challenge-2026/challenge_data

<div align="center"> <h1 style="font-size: 32px;">PrimeBot Household Bimanual Manipulation Challenge Dataset</h1>

<img src="./challenge_teaser.jpg" width="800" alt="PrimeBot Household Bimanual Manipulation Challenge">

中文 | English </div>

中文

目录

• —关于我们
• —更新日志
• —真机遥操作数据
• —训练集说明
• —验证集说明
• —数据集字段说明
• —URDF
• —图像
• —语言指令
• —本体感知与动作
• —机器人推理接口
• —UMI数据
• —数据概览
• —目录结构
• —数据集字段说明
• —图像
• —本体感知与动作
• —索引字段
• —标注与 IMU

关于我们

我们来自上纬新材-启元研究院，我们的使命是加速个人机器人时代到来，加速家用机器人时代到来。我们开源高质量面向家庭操作的双臂操作数据集，同时开放机器人硬件描述以供可视化、可复现研究。

如果本数据集对您的工作有帮助，感谢引用：

@misc{xu2026scalingbimanualhouseholdmanipulation,
      title={Scaling Bimanual Household Manipulation from 1,500 hours of Demonstrations to On-Policy Corrections}, 
      author={Jiafeng Xu and Qi Li and Yan Shen and Yiyu Ren and Travis Davies and Shaowen He and Ze Wang and Yifan Yang and Ran Cheng and Hao Dong},
      year={2026},
      eprint={2609.03591},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2609.03591}, 
}

如果您对挑战赛、数据集或其他未尽事项有任何疑问，可通过邮件联系：

likai@qiyuan-robot.com, xujiafeng26@stu.pku.edu.cn

更新日志

• —2026.09.08：第二批 100 小时UMI数据开源
• —2026.08.10：第一批 5 小时UMI数据开源
• —2026.08.06：全部 500+ 小时真机数据开源

真机遥操作数据

真机数据由启元机器人提供，数据集均以标准的 LeRobot V2.1 格式构造，示例Dataset代码，

pip install "lerobot==0.3.3" "mmengine==0.10.7" "torch==2.7.0" "numpy==1.26.4" "torchcodec==0.5" "torchvision==0.22.0"

python dataloader/custom_lerobot_dataset.py

训练集说明

训练集覆盖超过12种真实的家庭场景双臂操作任务，所有数据均包含精确到帧的语言标注，部分任务列表如下 | 任务编号 | 任务描述 | | :--- | :--- | | 1 | Use the gripper to fully open the washing machine door. | | 2 | Close the washing machine door tightly with the gripper. | | 3 | Put these two pieces of clothing into the washer. | | 4 | Take the clothing out of the washer and put it in the basket. | | 5 | Pick up the laundry basket with both grippers. | | 6 | Put the dirty clothes basket on the ground. | | 7 | Pick up the clothing and put it on the sofa. | | 8 | Put the clothing in the folding area. | | 9 | Unfold the clothing and fold it neatly. | | 10 | Place the folded clothing in the storage area. |

训练集分4个批次，互相独立不重复，可自由组合数据集训练策略， | 数据集名称 | 采集批次 | 任务类型 | 语言标注 | | :--- | :--- | :--- | :--- | | fulltaskbatch1train | 1 | 1-10 | 帧级切分 | | foldclothbatch2train | 2 | 8~10 | 帧级切分 | | partialtaskbatch3train | 3 | 1,4,5 | 帧级切分 | | partialtaskbatch4train | 4 | 1,2,3,9 | 整段标注 |

验证集说明

验证集与训练集格式完全一致，为防止策略过拟合到state上，有以下两点特殊处理，

• —验证集中observation.state数据含有随机噪声
• —验证集中action字段被全部置零

考虑到参赛团队算力资源与测试资源，本次挑战赛在不超过以下4个任务上进行线上和线下评测， | Task ID | Task Description | | :--- | :--- | | 1 | Use the gripper to fully open the washing machine door. | | 2 | Close the washing machine door tightly with the gripper. | | 3 | Put these two pieces of clothing into the washer. | | 9 | Unfold the clothing and fold it neatly. |

线上测评

1. 1.参赛者需要在validationdata的全部episode上做action预测，注意数据集中action为89维，线上/线下均只需要预测25维（定义见下文的`机器人推理接口`）即可，validationdata为
2. 2.fulltaskbatch1noisevalid
3. 3.foldclothbatch2noisevalid
4. 4.线上测评只提交预测结果的文件即可，提交入口为Google Form。每支队伍可多次提交，多次提交需要保持队伍信息不变，我们会在测评前取最新提交结果。
5. 5.提交文件的数据格式参考YourTeamName.zip，注意只需提交动作预测结果即可，标注、图像、本体状态等数据无需提交。
6. 6.线上评估分数与动作预测误差的 MSE(Mean Squared Error) 和 MAE(Mean Absolute Error) 相关，动作预测越准确，分数越高，分数前5名入围线下测评。
7. 7.测评时间在2026年8.26、8.28、8.31、9.2、9.4、9.7、9.9、9.11，在当日北京时间 10:00 AM 对提交结果统一测评，并于当天更新Leaderboard。

线下测评

1. 1.线下测评需要提交参赛模型的最终docker，提交形式另行通知入围队伍。
2. 2.我们提供的base docker包含了推理框架和机器人运行环境，入围队伍需在此基础上构建完整docker，包含模型ckpt、运行代码、模型依赖，
3. 3.其他详见下文的机器人推理接口，有任何疑问请联系 likai@qiyuan-robot.com, yan790@pku.edu.cn.

Q&A (Updating)

1. 1.“全部episode上做action预测” 是否表示两个指定 validation 数据集中的每个episode、每一帧都需要输出 action，并保持原始 30 FPS 和 episode 长度？

• —两个validation数据集中，每条episode，每一帧都要输出25维action，需要保持episode长度，但我们只在选中的instruction段计算分数。

1. 1.线上评分使用什么指标？89维字段是否全部计分？

• —线上评估分数与动作预测误差的 MSE 和 MAE 相关。只在25维上计算分数，提交的时候也只需要提交25维action预测。

1. 1.入围真机阶段后是否允许更新模型权重？真机推理的 action chunk 长度 N、控制频率和计算资源限制分别是多少？

• —入围后允许更新权重。真机推理action_chunk自定，参考下文的机器人推理接口。机器人控制的下发频率为30Hz，线下计算资源为4090机器，显存48GB，内存64GB。

1. 1.分数计算示例

• —某条episode长度为1000frames，共分4段（segment内均为左闭右开），如下所示，最终提交的action预测长度为1000，维度为25，我们会在[99, 420），[420, 910)这两段上计算分数。

数据集字段说明

URDF

我们开放机器人硬件描述以支持非商业用途的教学、实验、研究等项目。可从以下地址获取URDF：challenge_data/robot_description/x2w.urdf。

图像

包含三视角RGB图像，分辨率为1280*720，帧率30FPS，字段定义如下 | 数据集字段 | 数据源 | | :--- | :--- | | observation.images.x2wcameraheadrealsensecompressed | 头部相机 | | observation.images.x2wcameraleftwristzedxonegsrgbrawimagecompressed | 左手相机 | | observation.images.x2wcamerarightwristzedxonegsrgbrawimagecompressed | 右手相机 |

语言指令

数据集全部标注信息见${datasetname}/meta/info.json。每段语言标注均为左闭右开，即[startframeindex, endframe_index），以一条1000帧的操作任务为例说明如下 | 分段 | 片段1 | 片段2 | 片段3 | 片段4 | | :--- | :--- | :--- | :--- | :--- | | 索引 | 0~99(exclude) | 99~420 | 420~910 | 910~1000 | | 标注 | Start remote operation. | Open the washing machine door. | Close the washing machine door. | End remote operation. |

本体感知与动作

包括机器人状态信息（observation.state）与动作信息（action），维度均为89维，定义如下

1. 关节位置（Joint Position，索引 0-21）

2. 关节速度（Joint Velocity，索引 22-43）

3. 关节力矩（Joint Effort，索引 44-65）

4. 末端执行器位姿（Gripper Pose，索引 66-79）

5. 轮子关节状态（Wheel Joint State，索引 80-88）

机器人推理接口

我们提供了包含机器人运行环境的基础Docker镜像，参赛者需要实现自己模型的两个函数

def load_model(self):
    # TODO: implement model loading
    pass

def predict(self, sample: dict) -> np.ndarray:
    # TODO: implement inference, return np.ndarray of shape (N, 25)
    pass

def inference_loop(self):
    last_iner_time = time.time()
    while self.running:
        if self.prepare_in_progress or not self.enable_inference:
            time.sleep(0.1)
            continue

        tic = time.time()
        sample = self.update_input()
        prediction = self.predict(sample)

数据集中虽然记录了完整的action字段，但在推理中我们只选择机器人关节位置（25维）作为控制指令，完整字段定义如下

UMI数据

UMI数据由上海朗智格机器人科技有限公司提供，构造为标准的 LeRobot v2.1 数据集，包含 5 个双手操作 episode，以及左、右手腕载 ego 相机数据。

pip install "lerobot==0.3.3"

已在 Python 3.11、LeRobot 0.3.3、PyTorch 2.7.1、TorchVision 0.22.1 和 pyav 视频后端下完成加载与抽帧验证。

from lerobot.datasets.lerobot_dataset import LeRobotDataset

dataset = LeRobotDataset(
    repo_id="local/umi_sample_data_v21",
    root="/path/to/umi_sample_data_v21",
)

数据概览

数据集共 5940 帧、4 个唯一任务，帧率为 30 FPS，ego 视频分辨率为 960 × 960。

目录结构

umi_sample_data_v21/
├── data/chunk-000/                       # 5 个 episode Parquet 文件
├── videos/chunk-000/
│   ├── observation.images.left_ego/      # 左手 ego 视频
│   └── observation.images.right_ego/     # 右手 ego 视频
├── meta/
│   ├── info.json                         # 数据集与字段定义
│   ├── tasks.jsonl                       # 任务与 task_index 映射
│   ├── episodes.jsonl                    # episode 长度与任务
│   ├── episodes_stats.jsonl              # 每个 episode 的统计量
│   └── calibration.json                  # 相机与 IMU 标定参数
├── annotation/                           # episode 级任务和动作分段标注
└── imu/                                  # 左右手 IMU 数据

数据集字段说明

图像

本体感知与动作

observation.state 和 action 均为 16 维，字段顺序一致：

observation.state 表示当前帧状态。除末帧外，action[t] = state[t+1]；末帧 action 保留原始采集序列的下一时刻目标，因此不一定等于本 episode 的末帧 state。左右手位姿使用独立坐标系，不能直接计算双手之间的相对距离或姿态。

索引字段

标注与 IMU

标注区间采用 [start_frame_index, end_frame_index)，即包含起始帧、不包含结束帧。

English

Contents

• —About Us
• —Change Log
• —Real-World Teleoperation Data
• —Training Set Description
• —Validation Set Description
• —Dataset Field Description
• —URDF
• —Camera Images
• —Language Instructions
• —Proprioceptive and Actions
• —Robot Inference Interface
• —UMI Data
• —Dataset Overview
• —Directory Structure
• —Dataset Fields
• —Camera Images
• —Proprioception and Actions
• —Index Fields
• —Annotations and IMU

About Us

We come from PrimeBot Research Institute, Swancor Advanced Materials Co., Ltd. Our mission is to accelerate the advent of the era of personal robots and home robots. We open-source a high-quality, home-oriented bimanual manipulation dataset, and also provide open-source robot hardware descriptions for visualization and reproducible research.

If this dataset is helpful to your work, please cite it:

@misc{xu2026scalingbimanualhouseholdmanipulation,
      title={Scaling Bimanual Household Manipulation from 1,500 hours of Demonstrations to On-Policy Corrections}, 
      author={Jiafeng Xu and Qi Li and Yan Shen and Yiyu Ren and Travis Davies and Shaowen He and Ze Wang and Yifan Yang and Ran Cheng and Hao Dong},
      year={2026},
      eprint={2609.03591},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2609.03591}, 
}

If you have any questions about the challenge, dataset, or other information, feel free to drop an email:

likai@qiyuan-robot.com, xujiafeng26@stu.pku.edu.cn

Change Log

• —2026.09.08: Second batch of 100 hours of UMI data released open source
• —2026.08.10: First batch of 5 hours of UMI data released open source
• —2026.08.06: All 500+ hours of real robot data released open source

Real-World Teleoperation Data

Teleoperation data is sponsored by PrimeBot. The dataset is constructed in the standard LeRobot V2.1 format. Example code for loading the dataset:

pip install "lerobot==0.3.3" "mmengine==0.10.7" "torch==2.7.0" "numpy==1.26.4" "torchcodec==0.5" "torchvision==0.22.0"

python dataloader/custom_lerobot_dataset.py

Training Set Description

The training set covers more than 12 real-world dual-arm manipulation tasks in household scenarios. All data includes frame-accurate language annotations. Partial task list:

The training set consists of four independent, non-overlapping subsets. You can freely combine them for different training strategies. | Dataset Name | Collection Batch | Task Type | Instruction | | :--- | :--- | :--- | :--- | | fulltaskbatch1train | 1 | 1-10 | frame-level | | foldclothbatch2train | 2 | 8~10 | frame-level | | partialtaskbatch3train | 3 | 1,4,5 | frame-level | | partialtaskbatch4train | 4 | 1,2,3,9 | segments-level |

During the online assessment, participants must submit all predicted action trajectories on the validation_data dataset (scores are only calculated for the tasks mentioned above). The assessment dataset is:

• —fulltaskbatch1noisevalid
• —foldclothbatch2noisevalid

Validation Set Description

The validation set follows exactly the same format as the training set. To prevent policies from overfitting to state data, two special adjustments are applied:

• —The observation.state data in the validation set contains random noise.
• —All action fields in the validation set are set to zero.

Considering the computing resources of participating teams, the evaluation of this challenge will be conducted on no more than the follow 4 tasks: | Task ID | Task Description | | :--- | :--- | | 1 | Use the gripper to fully open the washing machine door. | | 2 | Close the washing machine door tightly with the gripper. | | 3 | Put these two pieces of clothing into the washer. | | 9 | Unfold the clothing and fold it neatly. |

Online evaluation

1. 1.Participants need to predict actions across all episodes of the validationdata. Note that the dataset contains 89 dimensions for actions; however, only 25 dimensions need to be predicted for both online and on-site evaluation participation (see `Robot Inference Interface` for definition). The validationdata consists of:
2. 2.fulltaskbatch1noisevalid
3. 3.foldclothbatch2noisevalid

1. 1.For online evaluation, only the file containing the prediction results needs to be submitted. The submission portal is Google Form. Each team can submit multiple times, ensuring that the team information remains unchanged. We will use the latest submission before the evaluation.

1. 1.The data format for submitted files should refer to YourTeamName.zip. Note that only action prediction results are required; annotations, images, ontology state, etc., are not needed.

1. 1.Online evaluation scores are related to the Mean Squared Error (MSE) and Mean Absolute Error (MAE) of action prediction errors. The more accurate the action prediction, the higher the score. The top 5 teams will advance to the on-site evaluation.

1. 1.The evaluation will be conducted on August 26, 28, 31, September 2, 4, 7, 9, and 11, 2026. The submitted results will be evaluated uniformly at 10:00 AM Beijing time on the same day, and the Leaderboard will be updated on the same day.

On-site evaluation

1. 1.For the on-site evaluation, teams need to submit the final Docker image of their participating model. The submission method will be notified to the shortlisted teams separately.

1. 1.The base docker we provide includes the inference framework and the robot's runtime environment. Shortlisted teams must build a complete Docker container on top of this, including the model (ckpt), runtime code, and model dependencies.

1. 1.For further details, please see Robot Inference Interface. For any questions, please contact likai@qiyuan-robot.com, yan790@pku.edu.cn.

Q&A (Updating)

1. 1.Does "Perform action prediction on all episodes" mean that every episode and every frame in the two specified validation datasets needs to output an action while maintaining the original 30 FPS and episode length?

• —For each episode and every frame in the two validation datasets, a 25-dimensional action needs to be output, maintaining the episode length, but we only calculate scores for the selected instruction segments.

1. 1.What metrics are used for online scoring? Are all 89 dimensions scored?

• —The online evaluation score is related to the MSE and MAE of the action prediction error. Scores are only calculated on the 25 dimensions, and only the 25-dimensional action prediction needs to be submitted.

1. 1.Is it allowed to update model weights after being shortlisted for the real machine stage? What are the action chunk length N, control frequency, and computational resource limits for real machine inference?

• —Weight updates are allowed after being shortlisted. The action_chunk for real machine inference is customizable; refer to Robot Inference Interface. The robot control sends commands at a frequency of 30Hz, and the on-site computing resources consist of a 4090 machine with 48GB of VRAM and 64GB of RAM.

1. 1.Score Calculation Example

• —An episode is 1000 frames long, divided into 4 segments (each segment is left-closed and right-open), as shown below. The final submitted action prediction is 1000 frames long with a dimension of 25. We will calculate the score on the segments [99, 420) and [420, 910).

Dataset Field Description

URDF

We are releasing the robot hardware description to support non-commercial projects such as teaching, experimentation, and research. The URDF can be obtained from the following address: challenge_data/robot_description.

Camera Images

The dataset includes three-view RGB images with a resolution of 1280×720 at 30 FPS. Field definitions:

Language Instructions

The complete annotation information for the dataset can be found in ${dataset_name}/meta/info.json. Each language annotation is left-closed and right-open, i.e., [startframeindex, endframeindex). Taking a 1000-frame manipulation task as an example:

Proprioceptive and Actions

Including robot state (observation.state) and action (action), both with 89 dimensions, defined as follows:

1. Joint Position (Index 0–21)

2. Joint Velocity (Index 22–43)

3. Joint Effort (Index 44–65)

4. End-Effector (Gripper) Pose (Index 66–79)

5. Wheel Joint State (Index 80–88)

Robot Inference Interface

We provide a base Docker image for the robot's inference environment, in which you will need to implement two functions for your own model

def load_model(self):
    # TODO: implement model loading
    pass

def predict(self, sample: dict) -> np.ndarray:
    # TODO: implement inference, return np.ndarray of shape (N, 25)
    pass

def inference_loop(self):
    last_iner_time = time.time()
    while self.running:
        if self.prepare_in_progress or not self.enable_inference:
            time.sleep(0.1)
            continue

        tic = time.time()
        sample = self.update_input()
        prediction = self.predict(sample)

Although the full action fields are recorded in the dataset, only robot joint positions are used as control commands(25-dimensional) during inference. The complete field definitions:

UMI Data

UMI data is sponsored by crobotia. The dataset is constructed in the standard LeRobot v2.1 dataset with five bimanual manipulation episodes and left/right wrist-mounted ego-camera data.

pip install "lerobot==0.3.3"

The dataset has been successfully loaded and frame-tested with Python 3.11, LeRobot 0.3.3, PyTorch 2.7.1, TorchVision 0.22.1, and the pyav video backend.

from lerobot.datasets.lerobot_dataset import LeRobotDataset

dataset = LeRobotDataset(
    repo_id="local/umi_sample_data_v21",
    root="/path/to/umi_sample_data_v21",
)

Dataset Overview

The dataset contains 5,940 frames and four unique tasks. All episodes are recorded at 30 FPS, and the ego videos have a resolution of 960 × 960.

Directory Structure

umi_sample_data_v21/
├── data/chunk-000/                       # Five episode Parquet files
├── videos/chunk-000/
│   ├── observation.images.left_ego/      # Left-hand ego videos
│   └── observation.images.right_ego/     # Right-hand ego videos
├── meta/
│   ├── info.json                         # Dataset and feature definitions
│   ├── tasks.jsonl                       # Task-to-task_index mapping
│   ├── episodes.jsonl                    # Episode lengths and tasks
│   ├── episodes_stats.jsonl              # Per-episode statistics
│   └── calibration.json                  # Camera and IMU calibration
├── annotation/                           # Episode-level and action-step annotations
└── imu/                                  # Left/right IMU data

Dataset Fields

Camera Images

Proprioception and Actions

Both observation.state and action are 16-dimensional and use the same field order:

observation.state represents the current-frame state. Except for the final frame, action[t] = state[t+1]. The final action retains the next-step target from the original capture sequence and therefore may differ from the final state of the episode. The left and right poses use independent coordinate systems and cannot be used directly to compute the relative distance or pose between the two hands.

Index Fields

Annotations and IMU

Annotation intervals use the half-open convention [start_frame_index, end_frame_index): the start frame is included and the end frame is excluded.