DynamicIntelligence/humanoid-robots-training-dataset

Dynamic Intelligence — Humanoid Robot Training Dataset

A first-person (egocentric) video dataset of human hand manipulation, designed for training humanoid robot policies via imitation learning. Each episode captures a person performing an everyday household task — folding clothes, moving dishes, opening doors — filmed from a head-mounted iPhone using its built-in LiDAR and depth sensors.

The dataset pairs each video with frame-level 3D hand tracking and camera pose data, giving learning algorithms both the visual input and the corresponding spatial trajectories they need to reproduce the demonstrated behavior on a robot.

How it works

Recording setup. A person wears an iPhone 13 Pro on their head (using a head mount). The phone runs the Record3D app, which simultaneously captures:

• —RGB video at 30 FPS
• —Depth maps via the LiDAR sensor
• —6-DoF camera pose from ARKit (position + orientation of the phone in the room)

Processing pipeline. After recording, each episode goes through an offline pipeline:

1. 1.Hand detection — MediaPipe detects 2D hand landmarks in every RGB frame
2. 2.3D reconstruction — The 2D landmarks are projected into 3D space using the corresponding depth map, producing real-world XYZ positions (in cm) relative to the camera
3. 3.Action computation — Frame-to-frame deltas are computed for both the camera and hand positions, representing the "actions" a robot would need to take

Result. Each episode contains a synchronized video and a parquet file with per-frame 3D observations and actions, formatted for the LeRobot framework.

Dataset overview

Tasks

What's in the data

Each episode has two files: a video (.mp4) and a parquet table with one row per tracked frame.

Observations (what the robot sees)

Actions (what the robot should do)

Metadata columns

Coordinate system

All 3D positions are relative to the camera:

• —X → right
• —Y → down
• —Z → forward (into the scene)

Hand values of [0, 0, 0] mean the hand was not detected in that frame (e.g. out of view or occluded).

File structure

├── data/
│   ├── chunk-000/          # Parquet files for episodes 0–99
│   └── chunk-001/          # Parquet files for episodes 100–144
├── videos/
│   ├── chunk-000/rgb/      # MP4 videos for episodes 0–99
│   └── chunk-001/rgb/      # MP4 videos for episodes 100–144
├── meta/
│   ├── info.json           # LeRobot dataset config
│   └── stats.json          # Column statistics (min/max/mean/std)
└── README.md

Quick start

With LeRobot

from lerobot.common.datasets.lerobot_dataset import LeRobotDataset

dataset = LeRobotDataset("DynamicIntelligence/humanoid-robots-training-dataset")

sample = dataset[0]
print(sample["language_instruction"])        # "Fold the t-shirt on the bed."
print(sample["observation.camera_pose"])     # tensor of shape [6]
print(sample["action.left_hand_delta"])      # tensor of shape [9]

Direct download

import pandas as pd
from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="DynamicIntelligence/humanoid-robots-training-dataset",
    filename="data/chunk-000/episode_000000.parquet",
    repo_type="dataset",
)
df = pd.read_parquet(path)
print(f"{len(df)} frames")
print(df[["timestamp", "observation.camera_pose", "language_instruction"]].head())

Visualizer

Browse episodes interactively: [DI Hand Pose Sample Dataset Viewer](https://huggingface.co/spaces/DynamicIntelligence/dynamic_intelligence_sample_data)

The viewer shows the egocentric video alongside time-series plots of camera pose and hand positions, so you can see exactly what the person was doing and how the tracking data aligns with the video.

Citation

@dataset{dynamic_intelligence_2025,
  author = {Dynamic Intelligence},
  title = {Humanoid Robot Training Dataset: Egocentric Hand Manipulation Demonstrations},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/datasets/DynamicIntelligence/humanoid-robots-training-dataset}
}

Contact

Organization: Dynamic Intelligence Email: shayan@dynamicintelligence.company