EgoSteer/EgoSteer-RealWorld

EgoSteer Real-World Bimanual Teleoperation Dataset

<p align="center"> <a href="https://arxiv.org/abs/2607.09701"><img src="https://img.shields.io/badge/Paper-arXiv-b31b1b?style=for-the-badge&logo=arxiv&logoColor=white" alt="Paper"></a> <a href="https://egosteer.github.io/"><img src="https://img.shields.io/badge/Project-Page-1a73e8?style=for-the-badge&logo=googlechrome&logoColor=white" alt="Project Page"></a> <a href="https://huggingface.co/EgoSteer/models"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Model-Hugging%20Face-FFD21E?style=for-the-badge&labelColor=555555" alt="Model"></a> </p>

<p align="center"> <a href="https://github.com/egosteer/egosmith"><img src="https://img.shields.io/badge/EgoSmith-Data%20Pipeline-24292e?style=for-the-badge&logo=github&logoColor=white" alt="EgoSmith"></a> <a href="https://github.com/egosteer/robot-stack"><img src="https://img.shields.io/badge/Robot%20Stack-Teleop%20%26%20Deploy-2ea44f?style=for-the-badge&logo=github&logoColor=white" alt="Robot Stack"></a> <a href="https://github.com/egosteer/egosteer"><img src="https://img.shields.io/badge/EgoSteer-Model%20%26%20Training-6f42c1?style=for-the-badge&logo=github&logoColor=white" alt="EgoSteer"></a> </p>

<p align="center"> <img src="assets/Teaser.png" width="100%" alt="EgoSteer teaser"> </p>

EgoSteer-RealWorld is the real-robot dataset collected and used in EgoSteer: A Full-Stack System Towards Steerable Dexterous Manipulation from Egocentric Videos. It contains 54,454 teleoperated episodes (192 hours, 20.75 M frames) of bimanual dexterous manipulation across 193 tasks, recorded on a RealMan dual-arm robot with two Ruiyan dexterous hands and two RGB-D cameras (head and chest), with free-form English language instructions for every episode. In the EgoSteer system, this dataset grounds the manipulation priors that EgoSteer-3B-Base learned from 9.6K hours of egocentric human videos onto the RealMan embodiment, producing EgoSteer-3B-RealMan. The dataset is released in LeRobot v3 format.

At a glance

<table> <tr><td><b>Episodes</b></td><td>54,454 (train 54,261 / val 193)</td></tr> <tr><td><b>Tasks</b></td><td>193 (56 common + 137 long-tail, seven manipulation categories)</td></tr> <tr><td><b>Frames</b></td><td>20,750,677 per stream at 30 Hz (192.1 h)</td></tr> <tr><td><b>Episode length</b></td><td>median 321 frames (10.7 s); min 43, max 3,076</td></tr> <tr><td><b>Cameras</b></td><td>head RGB-D and chest RGB-D, 640×480</td></tr> <tr><td><b>Proprioception</b></td><td>74-dim <code>observation.state</code> and <code>action</code> (arm joints, hand joints, wrist poses, fingertip positions)</td></tr> <tr><td><b>Language</b></td><td>1–12 English instructions per episode (typically 2–3)</td></tr> <tr><td><b>Size</b></td><td>3.2 TB (depth 2.65 TB, RGB 510 GB, parquet 17 GB)</td></tr> <tr><td><b>Format</b></td><td>LeRobot v3 (<code>codebaseversion: v3.0</code>), <code>robottype: egosteerrealmanbimanual</code></td></tr> </table>

Robot and sensors

<p align="center"><img src="assets/robot-stack.png" width="100%" alt="Overview of the Robot Stack"></p> <p align="center"><em>Overview of the Robot Stack (paper Figure 3). It unifiedly supports teleoperation, policy inference and human-in-the-loop correction; the bottom-left shows the RealMan embodiment on which this dataset was collected.</em></p>

The RealMan embodiment consists of two 7-DoF RealMan RM75-6F arms (arm1 = left, arm2 = right) and two 6-DoF Ruiyan RY-H2 dexterous hands (hand1 = left, hand2 = right). It carries one head-mounted and one chest-mounted Intel RealSense D455 camera, giving two egocentric viewpoints; the head camera frame is the world frame of the dataset, and the chest camera is described relative to it.

For teleoperation, a pair of PsiBot SynGlove-Air gloves and Vive Trackers capture the operator's wrist poses and hand joint angles, which drive the arms through inverse kinematics (mink) and the hands through joint mapping. The tracker, arm solvers and arm controllers run at 100 Hz, the glove, hand solvers and hand controllers at 80 Hz, and the cameras at 30 Hz. Data are recorded at these native rates and resampled to 30 Hz in this release (see Time alignment). The complete stack, which also serves policy inference and human-in-the-loop correction with the same control nodes, is open source in robot-stack.

All joint names match the MuJoCo models shipped in robot-stack (assets/robot_mjcf, assets/ruiyan_hand_mjcf) character for character, so mj_name2id can index them directly. joint_linkN means "the joint that drives linkN".

Data collection

Within the kinematic limits of the RealMan embodiment we designed 193 semantically distinct tabletop manipulation tasks and collected roughly 300 randomized demonstrations (about one hour) for most of them. The tasks fall into two groups:

• —Common tasks (56): everyday manipulations that are readily achievable with the current hardware and sensors, with high teleoperation success rates.
• —Long-tail tasks (137): infrequent and physically challenging manipulations, such as contact-sensitive operations without tactile feedback, included to cover the dexterous manipulation space as completely as possible.

By motion characteristics and physical interaction, the tasks span seven categories:

<p align="center"><img src="assets/task_examples.png" width="90%" alt="Representative tasks across the seven categories"></p> <p align="center"><em>Representative task examples across the seven manipulation categories.</em></p>

The task names are listed in meta/tasks.parquet and act as categories (task_index on every frame); the language supervision comes from the per-episode instructions described in the next section.

<details> <summary>All 193 tasks with episode counts</summary>

</details>

Every trajectory was collected under strict requirements on randomness, diversity and quality. The tabletop is cluttered and unstructured rather than pre-arranged: tablecloths, object instances and initial configurations are randomized and no trajectory is scripted, so a task cannot be identified from the visual input alone and the policy has to align the language instruction with the physical action. Operators were instructed to teleoperate in a natural, human-like manner, so demonstrations of the same task differ substantially in their execution.

<p align="center"><img src="assets/dataset_statistics.png" width="90%" alt="Dataset statistics"></p> <p align="center"><em>Dataset statistics: word clouds and top-30 frequencies of nouns and verbs in the language annotations, task duration by the seven categories, and per-task duration for the 56 common and 137 long-tail tasks.</em></p>

Language annotations

Each episode carries multi-level language annotations in the episode-level instructions column. They were produced in three steps:

1. 1.Model annotation. Qwen3-VL-Flash watched the synchronized head and chest videos of the episode together with the task name and wrote three English descriptions at increasing granularity: a short verb–noun gist (L1), a description of the action and the manipulated objects naming the acting hand (L2), and a step-by-step sequence of the key functional actions (L3). The prompt is given below.
2. 2.Human verification. Annotators checked and corrected every description while working in Chinese, using the videos as reference.
3. 3.Back-translation. The verified Chinese text was translated back into English, which is what the dataset ships.

Because the wording passed through a vision-language model, human reviewers and two translations, individual instructions can be imprecise: the model may misread a scene, reviewers differ in expertise and attention, and translation can introduce ambiguity or shift a term (for example, Chinese chess pieces named by their characters). Instructions are consistent with the task and the video at the level of intent, but should not be taken as verbatim ground truth for every detail.

<p align="center"><img src="assets/teleop_episode.png" width="90%" alt="Dual-view image sequence and three-level annotations of one episode"></p> <p align="center"><em>Head and chest views of one trajectory with its three-level language annotations.</em></p>

<details> <summary>Language labeling prompt (Qwen3-VL-Flash)</summary>

**CRITICAL VISUAL CONTEXT & PRIOR (READ CAREFULLY):**
You are observing two synchronized egocentric videos (Head View: top, Chest View: bottom) of an agent performing a manipulation task. This output will be used as language instructions for robotic training.
1. **The Agent's Hand:** The moving entity is the agent's bare end-effector. It generally has a grey base and black fingers.
2. **COLORED TIPS WARNING:** The tips/pads of the fingers often have GREEN, ORANGE, or RED tape/markers on them. THESE ARE PART OF THE FINGERS. They are NOT separate tools.
3. **EMPTY-HANDED PRIOR:** The agent is operating empty-handed. NEVER describe the agent as holding or using a 'green-tipped tool', 'hot knife', 'pliers', or any handheld instrument.
4. **ABSOLUTE GROUND TRUTH (TASK ALIGNMENT):** The specific task is **[{task_name}]**. This task name is your absolute ground truth for interpreting WHAT is being manipulated (Objects) and HOW it is being manipulated (Verbs). You MUST use the exact nouns implied by the task name.
5. **GRAMMAR:** Your description must be in **simple present tense**. Write in fluent English, avoid awkward phrasing.

**OBJECTIVE:**
Describe the agent's actions (focusing strictly on hand-object interactions) in **simple present tense** by integrating information from both views into a single, unified description at three levels of detail. **Since this is for robot training, you MUST completely ignore all task-irrelevant items.**

**CONSTRAINTS:**
1. **Unified Description:** Provide ONE consolidated set of descriptions.
2. **Levels of Detail:**
   - **Level 1 (Gist):** A concise summary of the main action. Should always be a verb+noun phrase (i.e Ring a bell) (<20 words).
   - **Level 2 (Descriptive):** Main action + features and spatial layout of the **ACTIVELY MANIPULATED OBJECTS ONLY** (<40 words). **DO NOT list or describe any stationary background clutter.** In your description, list which hand is performing the action.
   - **Level 3 (Sequential):** The step-by-step temporal flow of key functional actions (<70 words). Include essential phases (reach -> manipulate -> release) but OMIT trivial micro-adjustments or hovering. List which hand is performing each action.
3. **Zero Subjects (Strict):** **Start every single sentence directly with a verb** (e.g., 'Reach for...', 'Grasp...'). DO NOT use subjects (e.g., 'The person', 'The robot', 'The hand', 'It').
4. **Strict Focus on Interaction (NO CLUTTER):** Focus ONLY on the objects being actively touched, moved, or interacted with (and their immediate targets/receptacles). **Completely IGNORE all irrelevant background items** (e.g., wipes, boxes, tubes, stands that are not part of the task). Never write phrases like 'other items remain unchanged' or 'in the background'.
5. **Vocabulary Restrictions:**
   - DO NOT output words like 'robot', 'mechanical arm', 'gripper', 'human', or 'finger'.
   - DO NOT output colors of the agent's hand/tips.
6. **Task Vocabulary (Verbs & Nouns):** Your choice of verbs AND target nouns must strictly align with the task **[{task_name}]**.
7. **Action Logic & Validation:** Focus on the actual state changes of the objects. Verify actual contact using both views.
8. **Object Disambiguation:** Use spatial descriptors (e.g., 'the topmost card') ONLY for task-relevant items to distinguish them from each other.
9. **Tense:** Use **Simple Present** tense (e.g., 'reach', 'grasp', 'slide').
10. **Spatial Description:** Use the camera frame as the reference.
   - Use **'upper', 'middle', 'lower'** to describe distance of objects on table. For example, 'the upper left of the table' refers to the far side of the table, and 'the lower left' refers to the near side.
   - Use 'left' 'right' to describe horizontal relationships
   - Use 'on', 'on top of', 'above', 'below' etc. to describe vertical relationships.

**OUTPUT FORMAT:**
1. [Level 1 Description]
2. [Level 2 Description]
3. [Level 3 Description]

**EXAMPLE (If Task Name is 'Draw_cards'):**
1. Draw playing cards.
2. Slide the top cards from a central deck with left hand to draw them to the lower part of the table.
3. Reach toward the central deck with left hand, press down on the topmost card, and slide it backward. Return to the deck, press on the next card, and slide it backward to complete the draw.

</details>

Splits

The `val` split contains exactly one episode per task, drawn at random: 193 episodes; all other 54,261 episodes are train. Validation episodes are stored at the end of the episode index range so that the standard LeRobot splits field can express them (val: 54261:54454); the split column in meta/episodes carries the same information.

Dataset structure

EgoSteer-RealWorld/
├── meta/
│   ├── info.json                              # features, fps, totals, splits
│   ├── stats.json                             # global per-feature statistics
│   ├── tasks.parquet                          # task_index <-> task name (193 rows)
│   └── episodes/chunk-000/file-00[0-5].parquet   # one row per episode: length, task, instructions, calibration, stats
├── data/chunk-00[01]/file-XXX.parquet         # frame-level state/action/world2cam/indices, 1,769 files
├── videos/
│   ├── observation.images.head/chunk-00[01]/file-XXX.mp4          # RGB, h264, 1,769 files
│   ├── observation.images.chest/...                               # RGB, h264, 1,769 files
│   ├── observation.images.head_depth/chunk-00[0-9]/file-XXX.mp4   # depth, HEVC 12-bit lossless, 9,342 files
│   └── observation.images.chest_depth/...                         # depth, 9,342 files
└── assets/                                    # figures used in this card

Each parquet file and each video file contains a set of complete episodes: no episode is split across two files. In addition, every video keyframe is placed at the start of an episode, so reading one episode means seeking to its start timestamp (videos/<key>/from_timestamp in meta/episodes) and decoding up to its end timestamp, without decoding the episodes stored before it in the same file.

Frame-level features

State and action layout (74 dims)

observation.state and action share one layout: the state holds the measured values, the action the commanded targets at the same instant.

Notes:

• —Wrist poses and fingertip positions ([26:74]) are computed by forward kinematics from the joint values, the robot model and the per-episode hand-eye calibration. This is the unified human-to-robot action representation used by EgoSteer: wrist translation, 6D wrist rotation and fingertip keypoints of both hands in the camera frame. As described in the paper (Appendix B), the wrist frame is shifted axially forward so that the wrist-to-fingertip scale matches human proportions.
• —6D rotation is the first two columns of the rotation matrix, flattened column-major: [R00, R10, R20, R01, R11, R21].
• —Hand channels are normalized driver values of the RY-H2 hand rather than joint angles: the thumb-rotation channel spans [0, 0.6], the other five [0, 1].

Episode-level metadata

meta/episodes/*.parquet has one row per episode with the standard LeRobot columns (episode_index, length, tasks, dataset_from_index, dataset_to_index, data/video file indices and timestamps, per-episode stats/*) plus:

The camera extrinsics (world2cam) are frame-level features (see above), not episode metadata.

Time alignment

All eleven raw streams (two RGB, two depth, arm and hand states and commands) were recorded with their own timestamps at their native rates. Each episode is resampled onto a uniform 30 Hz grid that starts when every stream is available and ends when the first stream stops. Every stream is sampled by nearest neighbour to the grid time, with no interpolation and no zero-order hold. This is exactly what the EgoSteer inference stack does at run time, so the train and deployment time offsets between images and joints follow the same distribution.

Video encoding

Depth

Depth is stored with LeRobot's built-in depth video pipeline: metric depth is log-quantized to 12 bits over [0.01, 10.0] m (shift 3.5) and the quantized values are encoded losslessly. Within the working range of 0.5–2 m the quantization step is 1.3–1.8 mm, below the sensor noise. The quantization parameters are stored in meta/info.json, and LeRobotDataset dequantizes automatically; choose the unit with depth_output_unit="mm" or "m". A value of 0 means invalid / no return.

Coordinate frames and calibration

• —World frame ≡ head camera frame (observation.camera.head_world2cam is the identity). All wrist poses and fingertip positions are expressed in it.
• —A_to_B matrices map points from frame A to frame B: p_base = cam_to_base @ p_cam. The hand-eye matrices come from an eye-to-hand calibration (OpenCV calibrateHandEye, PARK) and are published unchanged.
• —observation.camera.chest_world2cam = inv(chest_cam_to_left_base) @ head_cam_to_left_base (derived through the left arm), stored on every frame so that robot and human data share one layout.
• —Lengths are in metres, angles in radians, matrices row-major, points are column vectors.

Loading the dataset

from lerobot.datasets.lerobot_dataset import LeRobotDataset

# The full dataset is 3.2 TB. LeRobot downloads it into ~/.cache/huggingface/lerobot by default.
ds = LeRobotDataset("EgoSteer/EgoSteer-RealWorld", depth_output_unit="mm")

frame = ds[0]
frame["observation.images.head"]         # float32 [3, 480, 640] in [0, 1]
frame["observation.images.head_depth"]   # float32 [1, 480, 640], millimetres, 0 = invalid
frame["observation.state"]               # float32 [74]
frame["action"]                          # float32 [74]
frame["observation.camera.chest_world2cam"]   # float32 [16], world (head camera) -> chest camera, row-major 4x4
frame["task"]                            # task name

ep = ds.meta.episodes[frame["episode_index"].item()]
ep["instructions"]                       # list of English instructions for this episode
ep["calibration/head_intrinsics"]        # 9 floats, K row-major

# Only the validation episodes (one per task):
val = LeRobotDataset("EgoSteer/EgoSteer-RealWorld", episodes=list(range(54261, 54454)))

Tips:

• —Random single-frame access decodes four videos; the depth streams only have a keyframe at the start of each episode, so random access costs about a second per frame. Iterating an episode in order, or decoding episodes with PyAV directly from meta/episodes (videos/<key>/from_timestamp … to_timestamp), is fast.
• —To inspect the dataset before downloading the videos, fetch only the metadata and parquet files: hf download EgoSteer/EgoSteer-RealWorld --repo-type dataset --include "meta/*" "data/*" (17 GB).
• —Depth quantization and every encoding parameter are recorded under features.*.info in meta/info.json.

Training EgoSteer on this dataset

The EgoSteer training code reads WebDataset shards. Convert the release with the script shipped there, then fine-tune from EgoSteer-3B-Base:

python scripts/lerobot_to_wds.py --root /path/EgoSteer-RealWorld --out /path/EgoSteer-RealWorld.wds
python scripts/verify_wds.py    --wds  /path/EgoSteer-RealWorld.wds --root /path/EgoSteer-RealWorld

See `data/data.md` in that repository for the shard layout and the fine-tuning guide.

License

Released under the Apache License 2.0, the same license as the EgoSteer code and models.

Citation

@misc{zhong2026egosteerfullstacksteerabledexterous,
  title={EgoSteer: A Full-Stack System Towards Steerable Dexterous Manipulation from Egocentric Videos},
  author={Yifan Zhong and Zhang Chen and Tianrui Guan and Fanlian Zeng and Yuyao Ye and Tianjia He and Ka Nam Lui and Jiayi Li and Tingrui Zhang and Ruilin Yan and Xinhao Ji and Guangyu Zhao and Wenjie Lou and Jiayuan Zhang and Yuanpei Chen and Yaodong Yang},
  year={2026},
  eprint={2607.09701},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2607.09701},
}