MIT-Media-Lab/egotouch-annotations-v1-leftfix

egotouch-annotations-v1-leftfix

Annotations only. This repository does not contain images or video.

This release corrects the left-hand MANO rotation convention in `egotouch-annotations-v1`. It keeps the original episode structure, instructions, tactile data, and sample index.

Correction

The source pipeline generated left-hand joints by mirroring a right-hand MANO model. However, it stored the rotation matrices in the original right-hand convention. This mismatch gave incorrect MANO-local coordinates for the left hand.

Let M = diag(-1, 1, 1). This release applies these operations to left-hand episodes:

global_orient_camspace = M @ global_orient_camspace @ M
hand_pose[:, joint] = M @ hand_pose[:, joint] @ M
global_orient_worldspace = R_camera_to_world @ global_orient_camspace

Reflection conjugation keeps each corrected matrix in SO(3). The correction does not mirror joints or translations again.

Unchanged data

The following data is unchanged:

• —joints_camspace and joints_worldspace
• —transl_camspace, transl_worldspace, and wrist positions
• —tactile pressure grids
• —MANO shape parameters (beta)
• —camera intrinsics and extrinsics
• —RGB frame references (video_decode_frame)
• —instructions and paraphrases
• —episode membership and frame counts
• —all right-hand annotations

Files

egotouch.tar
  Annotation/egotouch/episodic_annotations/*.npy

episode_frame_index.npz
  index_frame_pair       (3,687,389, 2) uint32
  index_to_episode_id    (111,159,)

full_repair_audit.json
  source and corrected hashes, counts, and validation results

The index is byte-identical to the parent release. Each row of index_frame_pair gives an episode ordinal and a frame number.

Load one sample

Extract egotouch.tar. Then load a sample with this code:

import numpy as np

sample_id = 0
index = np.load("episode_frame_index.npz", allow_pickle=True)
episode_slot, frame_id = index["index_frame_pair"][sample_id]
episode_id = str(index["index_to_episode_id"][episode_slot])

path = f"Annotation/egotouch/episodic_annotations/{episode_id}.npy"
episode = np.load(path, allow_pickle=True).item()
rgb_frame_id = int(episode["video_decode_frame"][frame_id])

Each episode stores MANO parameters, camera data, world-space joints, camera-space joints, tactile grids, instructions, paraphrases, and absolute RGB frame indices. The anno_type field identifies the target hand for the episode.

Coordinate convention

The per-frame extrinsics matrices use the world-to-camera convention. The global_orient_camspace and global_orient_worldspace fields contain rotation matrices. The world frame stays fixed within each source recording.

Chunk-level anchor-camera conversion belongs in the VITRA data loader. This release does not bake an anchor-camera frame into the stored annotations.

Validation

The full release passed these checks:

• —all 111,159 archive members load successfully.
• —archive names and order match the parent release.
• —episode identifiers and frame counts match the index.
• —no index row points outside an episode.
• —the maximum rotation determinant error is 5.65e-7.
• —the maximum rotation orthogonality error is 6.34e-5.
• —the maximum camera-to-world orientation consistency error is 0.0.
• —sampled right-only episodes are byte-identical to the parent release.

The audit file contains the archive hashes and exact validation counts.

Compatibility

Use this release for training that derives left-hand MANO-local coordinates from stored rotations. Rebuild action caches that were made from the uncorrected left-hand rotation fields.

Caches that use only stored joints and tactile grids do not change numerically. Their training protocol can still require a new comparison run for reproducibility.

RGB frames

The video_decode_frame field indexes the source EgoTouch video. Get the video from the EgoTouch source release.

Integrity

egotouch.tar
fe49162862d792861612c6405eebba958197a30df519a6e0fe1aa3e1db3f3088

episode_frame_index.npz
f9908d8733f00382a8fd762a1c8ab6f4b34fa47c5603c512c29fa66e31f47343