CoolFace
Datasetpublic

basematrix/ego-desktop-demo

BaseMatrix EGO Desktop Demo Human hand manipulation demonstrations captured with binocular egocentric cameras + exocentric view, processed through a 6-stage pipeline into structured observations for robot learning. This is a preview dataset. Full-scale datasets with hundreds of episodes are available upon request — contact info@basematrix.ai. Quick Start from lerobot.common.datasets.lerobot_dataset import LeRobotDataset dataset =… See the full description on the dataset page: https://huggingface.co/datasets/basematrix/ego-desktop-demo.

sourceHugging Facecc-by-nc-4.0updated 1mo agoView on Hugging Face
0likes124downloads
Dataset Card

BaseMatrix EGO Desktop Demo

Human hand manipulation demonstrations captured with binocular egocentric cameras + exocentric view, processed through a 6-stage pipeline into structured observations for robot learning.

This is a preview dataset. Full-scale datasets with hundreds of episodes are available upon request — contact info@basematrix.ai.

Quick Start

python
from lerobot.common.datasets.lerobot_dataset import LeRobotDataset

dataset = LeRobotDataset("basematrix/ego-desktop-demo")
print(dataset)
print(dataset[0].keys())

Dataset Summary

PropertyValue
FormatLeRobot v3.0
Episodes1
Frames2,114
Duration~70 seconds
FPS30
SceneDesktop object manipulation
Capture deviceOrbbec EGO binocular (stereo fisheye, 1600x1300) + iPhone EXO (1920x1080)
Time syncHardware PTS (< 40 us L-R stereo) + audio beep sync (< 3.3 ms EXO)

Features (24 channels)

Observation — Vision (3 video streams)

FeatureShapeDescription
observation.images.ego_left(3, 1300, 1600)Left stereo fisheye, 30fps H.264
observation.images.ego_right(3, 1300, 1600)Right stereo fisheye, 30fps H.264
observation.images.exo_iphone(3, 1080, 1920)Exocentric iPhone view, 30fps

Observation — State (11 channels)

FeatureShapeDescription
observation.state.camera_pose(7,)ORB-SLAM3 stereo-inertial 6DoF: tx, ty, tz, qx, qy, qz, qw
observation.state.left_wrist(3,)Left wrist 3D position in world frame (meters)
observation.state.right_wrist(3,)Right wrist 3D position in world frame (meters)
observation.state.left_keypoints_3d(63,)Left hand 21 joints x 3 coords (MANO topology, world frame)
observation.state.right_keypoints_3d(63,)Right hand 21 joints x 3 coords (MANO topology, world frame)
observation.state.left_contact(1,)Left hand contact state (0/1)
observation.state.right_contact(1,)Right hand contact state (0/1)
observation.state.left_force(3,)Reserved — force/torque (zeros, sensor not connected)
observation.state.right_force(3,)Reserved — force/torque (zeros, sensor not connected)
observation.state.left_tactile(48,)Reserved — tactile array (zeros, sensor not connected)
observation.state.right_tactile(48,)Reserved — tactile array (zeros, sensor not connected)
observation.state.gripper_force(2,)Reserved — gripper force (zeros, sensor not connected)

Action (4 channels)

FeatureShapeDescription
action.left_wrist_delta(3,)Left wrist displacement (dx, dy, dz in meters)
action.right_wrist_delta(3,)Right wrist displacement (dx, dy, dz in meters)
action.left_grasp(1,)Left hand grasp signal (0 = open, 1 = grasping)
action.right_grasp(1,)Right hand grasp signal (0 = open, 1 = grasping)

Processing Pipeline

Each recording goes through 6 automated stages:

StageToolOutput
L0Audio beep detection (1kHz)Multi-camera time alignment
L1Nearest-neighbor PTS matchingStereo frame pairs + EXO sync
L2ORB-SLAM3 (stereo-inertial)6DoF camera trajectory
L3Hands23 + WiLoR + HaWoRContact detection, 3D hand skeleton, world-frame wrist trajectory
L4GPT-4.1 VisionSemantic annotation (objects, phases, sub-activities)
L5Cross-layer verification (24 checks)Quality score + dataset card
L6LeRobot v3.0 packagingThis dataset

Hand Tracking Stack

Three complementary models provide different levels of hand understanding:

  • —Hands23 (HOI-DETR): Hand-object interaction detection — bounding boxes, contact state, grasp classification
  • —WiLoR-mini: Single-frame 3D hand pose — 21 MANO keypoints per hand
  • —HaWoR: Temporal hand-in-world reconstruction — globally consistent wrist trajectories in ORB-SLAM3's world coordinate frame

Retargeting

This dataset provides human-frame observations (hand skeleton + wrist trajectory). To use with a specific robot embodiment:

  • —For parallel-jaw grippers (Franka, ALOHA, UR5): Map left/right_wrist + left/right_grasp to end-effector pose + gripper open/close via inverse kinematics
  • —For dexterous hands (LEAP, Allegro, Shadow): Map left/right_keypoints_3d (21 MANO joints) to robot finger joint angles via fingertip-position optimization

The action.{left,right}_wrist_delta features can serve as a starting point for action-space definition, though most VLA pipelines will want to define actions in robot joint space after retargeting.

Coordinate System

  • —World frame: ORB-SLAM3 stereo-inertial SLAM coordinate system. Origin at first tracked frame. Right-handed, Y-up.
  • —Hand keypoints: Relative to wrist (wrist = origin), following MANO joint topology (0=wrist, 1-4=thumb, 5-8=index, 9-12=middle, 13-16=ring, 17-20=pinky).
  • —Camera pose: Position (tx, ty, tz) in meters + orientation as unit quaternion (qx, qy, qz, qw).

Capture Hardware

  • —Head-mounted: Orbbec Femto Mega (EGO configuration) — 140g, dual global-shutter fisheye, 1000Hz IMU, 120mm stereo baseline, H.264 encoding
  • —Exocentric: iPhone — standard wide lens, 1080p 30fps, audio beep sync

Citation

bibtex
@misc{basematrix2026ego,
  title={BaseMatrix EGO Desktop Demo: Binocular Egocentric Hand Manipulation Dataset},
  author={BaseMatrix},
  year={2026},
  url={https://huggingface.co/datasets/basematrix/ego-desktop-demo}
}

License

CC BY-NC 4.0 — free for research and evaluation. Contact info@basematrix.ai for commercial licensing.

Contact