introvoyz042/egocentric-vr-capture-1h-multimodal-sample

Egocentric VR Capture — 1-Hour Multimodal Inspection Sample

13 real-world task episodes / 108,029 frames / approximately 60 minutes captured with Meta Quest 3. Each episode combines egocentric RGB and audio with synchronized headset, camera, body, and hand tracking in a LeRobot v3-style package.

<img src="assets/dataset_preview.gif" alt="EXYLOS XR dataset preview" width="1280">

This publicly accessible dataset is an inspection slice produced by the EXYLOS real-world data pipeline. It demonstrates capture quality, synchronization, schema, and QA metadata before a larger commercial delivery.

Use it / Skip it

Use it for

• —Evaluating real-world egocentric Meta Quest 3 capture.
• —Inspecting synchronized video, audio, head, body, and hand tracking.
• —Testing human-motion preprocessing, tracking masks, temporal annotations, retargeting, and ingestion.
• —Reviewing EXYLOS delivery structure and QA evidence.

Skip it if you need

• —Robot actions, torques, force, tactile, or contact measurements.
• —Depth, segmentation, or object-pose ground truth.
• —Outcome, success/failure, reward labels, or a declared closed action ontology.
• —Externally validated mocap accuracy or a training-scale benchmark.

This is human XR capture, not robot execution data. The frame data contains no robot-control action feature or robot-control signal.

At a glance

Verify the core numbers

import json
from huggingface_hub import hf_hub_download

repo = "ExylosAi/egocentric-vr-capture-1h-multimodal-sample"
path = hf_hub_download(repo, "meta/info.json", repo_type="dataset")
info = json.load(open(path, encoding="utf-8"))
print(info["robot_type"], info["fps"], info["total_episodes"], info["total_frames"])
# -> meta_quest 30 13 108029

Load it

Load the lightweight Viewer mirror:

from datasets import load_dataset

repo = "ExylosAi/egocentric-vr-capture-1h-multimodal-sample"
frames = load_dataset(repo, split="train")
print(frames.num_rows)  # 108029
print(frames[0]["observation.state"])  # 390-D tracking vector

Load the 13 episode videos:

videos = load_dataset(repo, "videos", split="train")
print(videos.num_rows)  # 13

Load temporal action and body-part annotation tables:

level2 = load_dataset(repo, "annotations_l2", split="train")
level3 = load_dataset(repo, "annotations_l3", split="train")
print(level2.num_rows, level3.num_rows)  # 680 1153

Download canonical Parquet and metadata without the 4.43 GB of video:

from pathlib import Path
from huggingface_hub import snapshot_download
import pyarrow.dataset as ds

root = Path(snapshot_download(
    repo, repo_type="dataset",
    allow_patterns=["data/**/*.parquet", "meta/**", "supplemental/**"],
))
frames = ds.dataset(root / "data", format="parquet").to_table()

What you get

• —RGB + audio: one head-mounted video per episode.
• —Head: headset and head-camera position plus quaternion.
• —Body: positions, rotations, and binary validity for 26 joints.
• —Hands: 21 joint positions per hand plus visibility flags.
• —Provenance: video PTS, video-frame index, and native pose-frame index.
• —Metadata: task, environment, pseudonymous operator, and calibration.
• —Annotations: episode captions plus timestamped action and body-part sub-action spans.
• —QA: dropout, held-state, sentinel consistency, and jitter diagnostics.

<details> <summary><b>390-D state breakdown</b></summary>

Body confidence is binary, not calibrated. Upper-body and hand joints are camera-based estimates. Lower-body joints are generated rather than directly sensed. </details>

<details> <summary><b>Task coverage</b></summary>

</details>

Temporal annotations

supplemental/annotations.json provides a three-level timeline for every episode. All timestamps are relative to the start of the episode.

The Level 2 timeline covers approximately 99.87% of the delivered duration without overlapping spans. Every Level 3 span is contained within its parent Level 2 segment. Level 2 and Level 3 are also exposed as the annotations_l2 and annotations_l3 Viewer configs.

{
  "episode_id": 0,
  "start_ms": 3500,
  "end_ms": 13250,
  "action": "spray mirror",
  "level3": [
    {
      "body_part": "right_hand",
      "sub_action": "sprays the mirror"
    }
  ]
}

Tracking and QA

All poses use a static, left-handed capture frame: +X right, +Y up, +Z forward; positions are meters and quaternions use (x, y, z, w).

Native poses are mapped to the 30 Hz video grid by nearest-sample selection without smoothing or interpolation. Repeated samples remain visible through source.pose_frame. Audio, video, and pose share one capture clock; audio/video offsets and durations were checked within 50 ms.

Lost hands use (0,0,0) for all 21 joints with .visible == 0; consumers must mask them. Zero body-flag dropout does not establish physical accuracy.

File layout

meta/             info, calibration, tasks, episode metadata and stats
supplemental/     annotations, delivery metadata and pose QA contract
data/             canonical synchronized frame Parquet
videos/           13 canonical head-RGB MP4 files
viewer_data/      flattened 108,029-row default Viewer config
viewer_index/     13-row episode index
viewer_videos/    13-row video Viewer config
viewer_annotations/ flattened Level 2 and Level 3 annotation tables
assets/           dataset preview GIF

Scaling up

This repository is a one-hour inspection slice. Larger commercial deliveries can add hours, operators, tasks, environments, modalities, denser or customer-specific temporal annotations, customer-defined QA thresholds, retargeting, and alternative schemas. Volume, rights, acceptance criteria, and supported modalities are scoped separately.

Notes and limitations

• —13 episodes and three operators, mostly household tasks.
• —One egocentric RGB view; no depth, segmentation, object state, force, torque, or tactile streams.
• —Temporal Level 2 and Level 3 spans are descriptive natural-language annotations; no outcome, reward, or success/failure labels are included.
• —Body and hands are estimated; lower-body pose is generated.
• —Absolute translation, rotation, drift, and retargeting accuracy were not measured against external ground truth.
• —Camera intrinsics vary by capture; lens-distortion coefficients are absent.
• —Not a benchmark and not sufficient by itself for robot-transfer or model-performance claims.
• —Real-world video and audio remain subject to the repository license and must not be redistributed or used outside the permitted terms.

License and access

This dataset is publicly accessible but proprietary. Public access does not grant rights to redistribute, publish extracts, train commercial models, create derivative datasets, sublicense, or deploy commercially. Permitted uses require a separate written agreement with EXYLOS.

Citation

If required by the governing agreement, cite the repository and the exact snapshot revision used.