ExylosAi/egocentric-vr-capture-20h-multimodal-sample

Egocentric VR Capture — 20-Hour Multimodal Inspection Sample

195 real-world task episodes / 2,283,482 frames / 21.14 delivered hours captured with consumer VR hardware. Each episode combines egocentric RGB and audio with synchronized headset, camera, body, and hand tracking in a LeRobot v3-style package.

This publicly accessible 20-hour-scale dataset is produced by the EXYLOS real-world data pipeline. Files and the Dataset Viewer can be accessed without individual approval; use remains subject to the proprietary evaluation license. It demonstrates multi-operator task coverage, capture structure, synchronization, hierarchical temporal annotations, calibration, and pose provenance before a larger commercial delivery.

Use it / Skip it

Use it for

• —Evaluating real-world egocentric consumer-VR capture at multi-hour scale.
• —Inspecting synchronized video, audio, head, body, and hand tracking.
• —Testing human-motion preprocessing, tracking masks, temporal annotations, retargeting, and ingestion.
• —Reviewing EXYLOS delivery structure, task coverage, and QA evidence.

Skip it if you need

• —Robot actions, torques, force, tactile, or contact measurements.
• —Depth, segmentation, or object-pose ground truth.
• —Outcome, success/failure, reward labels, or a declared closed action ontology.
• —Externally validated mocap accuracy or a clean benchmark without additional pose filtering.

This is human XR capture, not robot execution data. The frame data contains no robot-control action feature or robot-control signal.

At a glance

Verify the core numbers

import json
from huggingface_hub import hf_hub_download

repo = "ExylosAi/egocentric-vr-capture-20h-multimodal-sample"
path = hf_hub_download(repo, "meta/info.json", repo_type="dataset")
info = json.load(open(path, encoding="utf-8"))
print(info["robot_type"], info["fps"], info["total_episodes"], info["total_frames"])
# -> egocentric_capture 30 195 2283482

Load it

Coordinate convention: all poses in observation.state are expressed in the ROS 2 REP-103 convention — a right-handed, Z-up world with +X forward, +Y left, +Z up; positions in meters, quaternions ordered (x, y, z, w). headset and head_camera are body frames (x forward, y left, z up). The camera optical frame (x right, y down, z forward, the frame the intrinsics are calibrated in) is reached from the head_camera body frame by the fixed rotation published as body_to_optical in meta/calibration.json, alongside T_head_camera (camera body frame in the headset frame) and the pinhole intrinsics. A map → headset → head_camera → head_camera_optical TF tree built from these values, with a sensor_msgs/CameraInfo from the intrinsics, projects the hand joints onto the published video. supplemental/pose_contract.json → coordinate_frame records the convention and the exact basis change from the headset's native left-handed Y-up frame, which is lossless and invertible.

Viewer provenance: the default viewer_data/ mirror is published together with the canonical frame Parquet in the vendor-neutral media snapshot revision 8a095939c2df9d9fe26986e909b82cb730f8b8af. It preserves the canonical pose values and adds only task, environment, video-path, video-timestamp, and episode-duration columns. The videos config resolves repository-local media through @main, while the snapshot revision records the verified immutable payload. The episodes, annotations_l2, and annotations_l3 configs use the same unchanged metadata and annotations.

Stream the frame-level Viewer mirror without downloading the full table:

from datasets import load_dataset

repo = "ExylosAi/egocentric-vr-capture-20h-multimodal-sample"
frames = load_dataset(repo, split="train", streaming=True)
first = next(iter(frames))
print(len(first["observation.state"]))  # 390

Load the episode index and video rows:

episodes = load_dataset(repo, "episodes", split="train")
videos = load_dataset(repo, "videos", split="train")
print(episodes.num_rows, videos.num_rows)  # 195 195

The videos config is episode-oriented. When several episodes share one physical MP4, use clip_start_s and clip_end_s to resolve the delivered episode interval.

Load temporal action and body-part annotation tables:

level2 = load_dataset(repo, "annotations_l2", split="train")
level3 = load_dataset(repo, "annotations_l3", split="train")
print(level2.num_rows, level3.num_rows)  # 12399 23244

Download canonical Parquet and metadata without the video payloads:

from pathlib import Path
from huggingface_hub import snapshot_download
import pyarrow.dataset as ds

root = Path(snapshot_download(
    repo, repo_type="dataset",
    allow_patterns=["data/**/*.parquet", "meta/**", "supplemental/**"],
))
frames = ds.dataset(root / "data", format="parquet")

What you get

• —RGB + audio: one head-mounted stream, with 189 physical MP4 files covering 195 episode intervals.
• —Head: headset and head-camera position plus quaternion.
• —Body: positions, rotations, and binary validity for 26 joints.
• —Hands: 21 joint positions per hand plus visibility flags.
• —Provenance: video PTS, video-frame index, and native pose-frame index.
• —Metadata: task, family, environment, pseudonymous operator, and per-episode camera calibration (pinhole intrinsics, T_head_camera, body_to_optical).
• —Annotations: episode captions plus timestamped action and body-part sub-action spans.
• —Pose disclosure: sensing provenance, coordinate conventions, generated lower-body status, and validation limitations.

<details> <summary><b>390-D state breakdown</b></summary>

Body confidence is binary, not calibrated. Upper-body and hand joints are camera-based estimates. Lower-body joints are generated rather than directly sensed. </details>

<details> <summary><b>Task coverage</b></summary>

</details>

Temporal annotations

supplemental/annotations.json provides a three-level timeline for every episode. All timestamps are relative to the start of the delivered episode interval.

Tracking and QA

All poses share one static REP-103 world frame (right-handed, +X forward, +Y left, +Z up); positions are meters and quaternions use (x, y, z, w). Native poses are mapped to the 30 Hz video grid, and source indices remain available for synchronization checks. Audio, video, and pose share one capture clock.

Structural validation passed for all 195 episode IDs, data/video references, frame counts, annotation IDs, and Level 3 parent containment. The delivered frame total exactly matches the episode index. Aggregate state statistics contain no NaN or infinite values, and body-confidence and hand-visibility fields stay in the declared [0, 1] range.

File layout

meta/                info, calibration, tasks, episode metadata and stats
supplemental/        annotations, delivery metadata and pose contract
data/                canonical synchronized frame Parquet
videos/              189 canonical head-RGB MP4 files
viewer_data/         enriched 2,283,482-row post-repair Viewer mirror
viewer_index/        195-row episode index
viewer_videos/       195-row episode-oriented video config
viewer_annotations/  flattened Level 2 and Level 3 annotation tables

Scaling up

This repository is a 20-hour-scale sample. Larger commercial deliveries can add hours, operators, tasks, environments, modalities, customer-specific annotations, QA thresholds, retargeting, and alternative schemas. Volume, rights, acceptance criteria, and supported modalities are scoped separately.

Notes and limitations

• —195 episodes and six pseudonymous operators across 43 tasks.
• —One egocentric RGB view; no depth, segmentation, object state, force, torque, or tactile streams.
• —Temporal Level 2 and Level 3 spans are descriptive natural-language annotations; no outcome, reward, or success/failure labels are included.
• —Body and hands are estimated; lower-body pose is generated.
• —Absolute translation, rotation, drift, and retargeting accuracy were not measured against external ground truth.
• —The world origin is the headset's tracking origin, with z = 0 at the headset's floor estimate; that estimate can differ from the physical floor (e.g. outdoors), so subtract the episode's median foot-joint height where floor-referenced z is needed. See pose_contract.json → world_origin.
• —Camera calibration varies across episodes; lens-distortion coefficients are absent.
• —Not a benchmark and not sufficient by itself for robot-transfer or model-performance claims.
• —The repository is publicly accessible. Do not redistribute its real-world video or audio without permission under the proprietary license.

License and access

This dataset and its files are publicly accessible without individual access approval. It remains proprietary. Public file access does not grant rights to redistribute, publish extracts, train commercial models, create derivative datasets, sublicense, or deploy commercially. Permitted uses require a separate written agreement with EXYLOS.

Citation

If required by the governing agreement, cite the repository and the exact snapshot revision used.