RLinf/rlt-maniskill-PegInsertionSide-v1-400-succ

RLT ManiSkill Joint

Dataset Summary

rlt_maniskill_joint is a LeRobot-style dataset for joint-control Robot Learning Token (RLT) training on the ManiSkill peg insertion task.

It is designed for the RLinf + OpenPI pi05_rlt_joint pipeline and is used in three stages:

1. 1.OpenPI supervised fine-tuning (SFT) base policy training
2. 2.RLT Stage 1 RL-token training
3. 3.RLT Stage 2 online RL initialization and normalization

The dataset corresponds to the ManiSkill task:

• —Environment: PegInsertionSideWideClearance-v1
• —Control mode: pd_joint_delta_pos
• —Default instruction: insert the peg in the hole

This dataset is intended for joint-space vision-language-action training rather than end-effector action prediction.

Supported Tasks

• —Vision-language-action supervised learning
• —Joint-control policy learning
• —RLT pretraining / post-training
• —Simulation-to-real aligned OpenPI data formatting

Data Format

This dataset follows the LeRobot-style schema expected by RLinf's pi05_rlt_joint dataconfig.

Each sample contains:

• —image: main third-person RGB image
• —wrist_image: wrist RGB image
• —state: 9-dimensional proprioceptive state
• —actions: action chunk in joint-control format
• —prompt: language instruction

Observation

image

Main camera RGB frame.

• —Type: image
• —Expected raw shape: typically 384 x 384 x 3
• —Semantics: third-person scene view

wrist_image

Wrist camera RGB frame.

• —Type: image
• —Expected raw shape: may be stored as either HWC or CHW
• —In RLinf/OpenPI preprocessing it is converted to HWC uint8
• —Semantics: wrist-mounted close-up view

state

Robot proprioceptive vector.

• —Type: float32[9]
• —Semantics: first 9 Panda joint-position values (qpos[:9]) used by the RLT joint-control pipeline

Action

actions

Joint-control action chunk.

• —Type: float32[T, A]
• —Default training horizon: T = 10
• —Stored action width may be larger in some intermediate exports, but the canonical OpenPI/RLinf contract uses:
• —action horizon: 10
• —action dimension consumed by the model: 8

Semantics:

• —control mode: pd_joint_delta_pos
• —each action step is an 8D joint-control command used by the ManiSkill RLT pipeline

Language

prompt

Language instruction for the task.

Default normalized prompt:

insert the peg in the hole