MIT-Media-Lab/hot3d-annotations-v1

hot3d-annotations-v1

Annotations only — no images, no video.

VITRA-style hand episodes for HOT3D, with per-hand instructions and paraphrases.

Aria (colour) only — the Quest3 half is not here

HOT3D is captured with two devices. Project Aria has an RGB camera; Quest 3 uses the headset's own tracking cameras, which are monochrome. Measured colourfulness (mean |B−G| + |G−R|) is 22.5–47.8 on Aria frames and 0.64–1.11 on Quest3 frames, i.e. Quest3 is exactly R = G = B.

Every other domain in this collection is colour, so only the Aria half is published here: 126 of the 250 recordings. The Quest3 half is complete in the source release and can be added by anyone who wants it.

What we did

Episodes are ours. The source release ships whole recordings (median 3,853 frames), so we re-cut it with VITRA's method — speed minima of the 3D wrist in world space:

gaussian smooth (sigma=1.0) -> local speed minima in a fixed window (win=15, i.e. 0.5 s at 30 Hz) -> merge runs shorter than min_seg=16 -> pad 2 frames on each end

sigma and win are quantities in time, converted per source frame rate. Left and right hands are cut independently, with the other hand's motion ignored. The validity mask is the source's own per-frame kept_frames, which on HOT3D is a genuine mask: dtype bool, values only {False, True}, true on 74.1% of frames, per-sequence mean median 0.785 (range 0.171–0.992), and no sequence is constant (0 of 252 all-true, 0 all-false).

Instructions are ours. Two rounds, both with Qwen3.5-122B-A10B-FP8: round 1 captions 8 frames per episode with the palm's future trajectory drawn on them; round 2 checks the sentence belongs to that hand, strips same-hand references ("Rinse the right hand." -> "Rinse the hand.", because training already prepends Left hand: ... Right hand: ...), and writes 1-3 paraphrases.

Episodes with no instruction are not included. Round 1 returns N/A when an episode shows no object interaction. Those episodes are excluded from both the archive and the index, so every episode here has a usable instruction.

Hands are balanced: 7,684 left / 10,906 right, i.e. the left hand is 41% of episodes.

Train / val / test split

splits/ ships the split we train with. `test` owns whole videos: its 6 videos appear in neither train nor val. train and val share the remaining 120 videos and are separated at the episode level, so val measures held-out clips from seen videos, not held-out environments. Balanced on frames, not episode count.

train.txt / val.txt / test.txt are episode-id lists (train and val share videos, so a video-name list cannot express the split). test_videos.txt lists the 6 held-out videos so the isolation can be checked by hand.

Files

hot3d.tar                   ->  Annotation/hot3d/episodic_annotations/*.npy
episode_frame_index.npz        index_frame_pair (N,2) uint32 + index_to_episode_id (E,)
splits/{train,val,test}.txt    episode ids
splits/test_videos.txt         the 6 videos test owns
splits/meta.json               parameters + achieved shares + self-checks

index_frame_pair row number is the sample id: row r = (episode ordinal, frame within that episode). len(index_frame_pair) is the size of the training set.

import numpy as np
# tar -xf hot3d.tar
z = np.load("episode_frame_index.npz", allow_pickle=True)
ep_slot, frame_id = z["index_frame_pair"][sample_id]
eid = str(z["index_to_episode_id"][ep_slot])
d = np.load(f"Annotation/hot3d/episodic_annotations/{eid}.npy", allow_pickle=True).item()
rgb_frame_id = int(d["video_decode_frame"][frame_id])

Each .npy is a dict with video_name, video_decode_frame, intrinsics, per-frame extrinsics (world->camera), anno_type (which hand this episode is for), text, text_rephrase, a quality dict of per-frame masks carried over from the source, and a left/right dict holding beta, hand_pose, global_orient_worldspace, transl_worldspace, joints_worldspace, kept_frames. text[hand] = [(sentence, (0, T))] and text_rephrase[hand] = [([paraphrases...], (0, T))].

Getting the frames

video_decode_frame indexes the source video, which we do not redistribute. Get it from HOT3D — https://facebookresearch.github.io/hot3d/ (licence agreement required), then decode by index (we use decord; a self-maintained sequential counter drifts silently if the decoder ever skips a frame).

Resolve the video by its full `video_name` path, never by basename. The 250 source videos carry only 3 distinct basenames — 214-1.mp4 (×126), 1201-1.mp4 (×77), 1201-2.mp4 (×47). Any basename, sorted-order or fuzzy match collapses 126 different recordings onto one file. We made exactly this mistake in an earlier render: 12 sample images collapsed pixel-for-pixel into 3 groups matching the 3 basenames, while the source videos have different sizes and checksums.

Known limitations

• —Paraphrase count averages 1.96, not a fixed number. Past 3 the model starts inventing; a sentence with no prepositional phrase honestly supports only one or two.
• —HOT3D subjects wear markers on the hands, so the RGB hands are not bare hands (the same is true of ARCTIC in this collection). Not fixable downstream.
• —Intrinsics imply 1407×1407 while the Aria video is 1408×1408 — a 1 px difference. Use the video's native size, not (2cx, 2cy).
• —Verified: the index lists exactly the episodes that have an instruction, every episode's stored frame count matches its index rows, and no index entry points at a missing episode.

Revision — 2026-09-08 (re-cut)

This release replaces the previous one. The previous episodes contained invalid frames and should not be used.

A defect in our episode-cutting step let frames with kept_frames == False (invalid hand pose — all-zero or NaN wrist coordinates) stay inside published episodes. The validity mask was only used to keep a cut point from landing on an invalid frame; it did not constrain what a segment contained. Worse, a run of invalid frames could suppress cutting altogether, so the gap was swallowed into one long segment instead of being excluded.

Every episode here is now built from a run of consecutive valid frames, so kept_frames is all-True by construction — verified over the whole collection: 568,369 episodes / 16.5 M frames, zero `kept_frames == False`. Both instruction rounds were regenerated for the new segmentation.

Segment counts and episode ids therefore changed, and the index and splits were rebuilt:

Splits are video_test (test owns whole videos disjoint from train/val; train and val share the remaining videos and are split at the episode level), balanced on frames at 90/5/5, seed 1.

<!-- COLLECTIONTABLESTART --> | dataset | episodes | training samples | our contribution | size | HF | |---|---:|---:|---|---:|---| | EPIC-KITCHENS-100 | 149,570 | 4,019,534 | episodes + text | 8.70 GB | `epic30-annotations-v1` | | EgoTouch | 107,364 | 3,123,675 | episodes + text + tactile | 17.02 GB | `egotouch-annotations-v1` | | GigaHands | 70,486 | 2,266,087 | episodes + text | 2.89 GB | `gigahands-annotations-v1` | | Ego-Exo4D | 67,051 | 1,757,474 | text only | 4.09 GB | `egoexo4d-annotations-v1` | | OakInk2 | 29,058 | 1,052,924 | episodes + text | 1.56 GB | `oakink2-annotations-v1` | | TACO | 23,757 | 736,136 | episodes + text | 1.34 GB | `taco-annotations-v1` | | HOT3D | 18,805 | 619,680 | episodes + text | 1.51 GB | `hot3d-annotations-v1` | | ARCTIC | 12,610 | 425,796 | episodes + text | 0.85 GB | `arctic-annotations-v1` | | H2O | 5,845 | 200,332 | episodes + text | 0.40 GB | `h2o-annotations-v1` | | Tachin | 2,669 | 79,584 | episodes + text + tactile | 3.38 GB | `tachin-annotations-v1` | | total | 487,215 | 14,281,222 | | 41.7 GB | |

Something-Something V2 was dropped from the collection (12 fps against 30 fps everywhere else, so a 16-step action chunk spans 1.33 s instead of 0.53 s). The repository still exists but should not be used.

<!-- COLLECTIONTABLEEND -->