Datasetpublic

zeno-labs/zeno-app-household

ZenO App Household 7 first-person recordings of ordinary household work, with both hands tracked in three dimensions and in real metres, laid out in LeRobot v2.1. Episodes 7 Frames 101,205 at 30 fps, about 56.2 minutes Video observation.images.head, 1920x1440 State 7 floats, camera position and orientation Action 20 floats, both wrists and both grippers Coordinate frame ROS REP 103, X forward, Y left, Z up, metric Watch it first Open in… See the full description on the dataset page: https://huggingface.co/datasets/zeno-labs/zeno-app-household.

sourceHugging Facecc-by-nc-4.0updated 2mo agoView on Hugging Face

0likes51downloads

Dataset Card

ZenO App Household

7 first-person recordings of ordinary household work, with both hands tracked in three dimensions and in real metres, laid out in LeRobot v2.1.


Episodes	7
Frames	101,205 at 30 fps, about 56.2 minutes
Video	`observation.images.head`, 1920x1440
State	7 floats, camera position and orientation
Action	20 floats, both wrists and both grippers
Coordinate frame	ROS REP 103, X forward, Y left, Z up, metric

Watch it first

**Open in the LeRobot dataset visualiser**

Scrub any episode in the browser, with the wrist and gripper channels plotted under the video. Nothing to install.

Why the depth matters

These were recorded on an iPhone with a LiDAR sensor. Depth is measured rather than inferred, so hand positions are in real metres in a gravity aligned world frame, not in pixels and not up to an unknown scale. A trajectory can be compared between two recordings, or handed to a robot, without first solving for how big the scene was.

Hand keypoints come from MediaPipe Hands. Each keypoint is back projected through the LiDAR depth map and the camera pose that ARKit reports for that frame, so what you get is a position in the room rather than a position in the picture.

The action vector

Twenty numbers per frame, ten for each hand, in this order:

Index	Channel	Meaning
0 to 2	`l_pos_x/y/z`	left wrist position, metres
3 to 8	`l_rot6d_0..5`	left wrist orientation, first two columns of the rotation matrix
9	`l_gripper`	left hand aperture, 0 closed to 1 open
10 to 12	`r_pos_x/y/z`	right wrist position, metres
13 to 18	`r_rot6d_0..5`	right wrist orientation
19	`r_gripper`	right hand aperture

Rotation is carried as a 6D representation rather than a quaternion because it is continuous, which matters when a network has to regress it.

action.left_valid and action.right_valid are booleans. A hand that was not detected in a frame carries its last known pose, so check the flag rather than looking for a sentinel value.

About the gripper channel

It is hand aperture, the thumb to index distance divided by hand span, mapped through one fixed pair of bounds for every episode: 0.12 reads as closed and 0.66 as open. Those bounds are the 5th and 95th percentile of aperture measured across 25,011 detections and they are recorded in meta/info.json.

It is deliberately not a grasp label. Aperture says how open the hand is, not whether it closed on anything, and the two disagree often enough that treating one as the other would be wrong. If you need contact events, derive them and validate them yourself.

Quality

Every episode here passed four automatic checks before it was included:

Check	What it asks
Revisit drift	When the camera returns to a place it has been, does that place land where it did before
Bone length	Do the finger segment lengths stay constant frame to frame
Speed	Do the hands and camera move at speeds a person can produce
Inertial agreement	Does the phone's own motion sensor agree with the pose track, and with what time offset

Recordings that fail any of them are held back rather than deleted. The checks are conservative and a failure usually means the tracking slipped, not that the recording is worthless.

1 recording(s) were excluded from this build by those checks.

Quick start

bash

pip install lerobot

python

from lerobot.common.datasets.lerobot_dataset import LeRobotDataset

ds = LeRobotDataset("zeno-labs/zeno-app-household")
sample = ds[0]

sample["observation.state"]            # [x, y, z, qx, qy, qz, qw]
sample["action"]                       # 20 floats, see the table above
sample["observation.images.head"]      # [C, H, W] uint8

What this is not

—There is no robot in it. The action is where a person's hands went, not a joint command. Retargeting to a specific arm is left to you.
—Body pose is not included. Hands and head only.
—Hand detection is not perfect. Detection rates per episode run from 94 to 99 percent, and hands drop out when they leave frame, go behind an object, or go into water.
—Episodes may be republished. Pin a commit if you need strict reproducibility.

Consent and privacy

Every recording was contributed knowingly by a person who was paid for it, under terms that cover research and commercial use of the resulting data. Faces and other identifying detail are removed before a recording enters the pipeline.

Same capture setup and the same schema, different recordings: `zeno-labs/egostation-iphone-lidar-household-v1`. Neither replaces the other.

Licence

CC BY-NC 4.0, non commercial. For the full corpus or commercial terms, contact support@zen-o.xyz.