VR-VLA/VR-egodex-annotation-converted-v6.0

VR-egodex-annotation-converted-v6.0

EgoDex converted from LeRobot v2.1 into the Layer-1 v0.6.0 annotation schema, with per-clip narration included as language sidecars.

• —314,839 clips · 78,282,306 frames · 724.8 hours @ 30 fps · 129 tasks
• —100% narration coverage (1 sidecar per clip)
• —71 GB annotations + 2.3 GB narratives

Videos are NOT included. This release contains annotations and narration only. Source video lives in `griffinlabs/EgoDex-LeRobot-v3.0`; orig_id in the manifest (<task>/episode_NNNNNN) is the join key.

Provenance and licence

Derived from EgoDex (Apple) via griffinlabs/EgoDex-LeRobot-v3.0. Every clip's parquet metadata carries "license": "EgoDex (Apple; see dataset terms)". Users of this release are responsible for complying with the original EgoDex terms — this conversion does not grant rights beyond them.

Narration was generated by a fine-tuned 27B vision-language model and is redistributed here under the same terms.

Layout

manifest.parquet                                   one row per clip — the index
annotations/egodex/chunk-NNN/<task>__episode_NNNNNN.parquet
narratives/egodex/chunk-NNN/<task>__episode_NNNNNN.parquet    same relative path

Annotation parquet — 12 columns, one row per frame

File-level parquet metadata key clip holds a JSON blob: clip_id, source, orig_id, fps, width, height, n_frames, hand_format, fingers_schema, intrins [fx, fy, cx, cy], has_left, has_right, gravity_up, embodiment, task_name.

Conventions: camera_model: opencv, world_frame: frame0_camera, rotation_storage: matrix_3x3_row_major, cam_pose_direction: world_to_camera, translation_units: meters_approx.

Narrative sidecar — 12 columns, one row per segment

segment_id, clip_id, start_f, end_f, start_s, end_s, lh_text, rh_text, ego_text, text, rephrasings, gen_model.

[right hand] pick up domino and place it upright | [ego] look down

Per-hand fields are null (not "") when that tag is absent from the cue, so "no left hand in this cue" stays distinguishable from "empty caption".

Conversion notes

Three decisions that are silent if wrong, recorded so results are reproducible:

Frame convention. EgoDex observation.state.camera is camera→world in ARKit world. Here the clip is first re-expressed into the frame-0 camera frame (X' = T0⁻¹ · X), then the camera pose is inverted to world→camera. gravity_up rides the same transform — ARKit world up (0,1,0) mapped through R0⁻¹.

21-keypoint selection. EgoDex ships 28 joints per side. The 4 Metacarpal joints and 3 body joints (Shoulder, Arm, Forearm) are dropped, leaving wrist + 20, emitted as wrist, thumb1-4, index1-4, middle1-4, ring1-4, pinky1-4 where 1-4 = Knuckle, IntermediateBase, IntermediateTip, Tip. Row 0 (the wrist) is exactly (0,0,0).

Validity. ARKit emits every joint every frame with no occlusion signal, so valid_l and valid_r are 1.0 throughout except where a joint is non-finite.

Validation

Checked with a per-clip, self-calibrating gate suite:

• —wrist keypoint is exactly zero on every clip
• —all rotation matrices orthonormal to 1e-4 with det ≈ +1
• —joint ordering verified by bone-length stability against a random permutation of the same clip's own joints (measured ≥218× better than permuted; ~1× would indicate a permuted layout). A hand held in a fixed pose is skipped rather than failed — its permuted control is equally flat, so the test has no discriminative power there.
• —every manifest row resolves to a file on disk

Loading

import pyarrow.parquet as pq
man = pq.read_table("manifest.parquet")
row = man.to_pylist()[0]
ann = pq.read_table(row["annotation_path"])   # per-frame
nar = pq.read_table(row["narration_path"])    # per-segment