zhenyuxie-zhzh/hot3d

HOT3D EgoHOI clips for the RGB object-entity (OEE) model

Exactly the data that the single-branch + object-entity version of the EgoHOI Wan2.2-TI2V-5B fine-tune reads at training and evaluation time. Nothing else from the original working tree is included (no pose caches, depth, ViPE trajectories or routing boxes); the pseudotactile/ and pointmap/ add-ons serve the later versions (see their sections below).

Derived from Meta's HOT3D dataset (Aria clips). The HOT3D license and terms apply to every video, mask and trajectory here; this repository only redistributes a processed subset for research use within the project.

Contents

Per split and clip the model reads:

• —videos/<clip>.mp4 -- the egocentric RGB clip (480x480, 150 frames, upright orientation).
• —skeleton_videos/<clip>.mp4 -- the rasterized hand-skeleton conditioning video, frame-aligned with the RGB clip.
• —camera_traj1/<clip>.json -- per-frame camera intrinsics/extrinsics.
• —obj_mask/<clip>/<frame:06d>.png -- binary union silhouette of every visible ground-truth object, one PNG per frame, stored in the RAW mp4 orientation (the loader applies the same ROTATE_270 it applies to RGB frames: --object_mask_orientation raw).

Tar members are named <split>/<modality>/...; extracting every shard into one directory reproduces the HOT3D_EgoHOI/{train,val}/... layout the code expects.

Restore

mkdir -p /data/HOT3D_EgoHOI && cd /data/HOT3D_EgoHOI
hf download zhenyuxie-zhzh/hot3d --repo-type dataset --local-dir /data/hot3d_download
for t in /data/hot3d_download/train/*.tar /data/hot3d_download/val/*.tar; do tar -xf "$t"; done
(cd /data/hot3d_download && sha256sum -c SHA256SUMS)   # optional integrity check (run before extracting)
python /data/hot3d_download/tools/make_local_manifest.py \
  --root /data/HOT3D_EgoHOI \
  --manifest /data/hot3d_download/manifests/egohoi_rgb_manifest.jsonl \
  --output /data/HOT3D_EgoHOI/hot3d_geometry_pseudotactile_v1/manifests/egohoi_rgb_manifest.jsonl
mkdir -p <model_dir>/prompt_embeddings && cp /data/hot3d_download/assets/prompt_embeddings/egohoi_default_prompt.pt <model_dir>/prompt_embeddings/

The manifest rewrite is required: the loader checks that each egohoi_video resolves to <root>/<split>/videos/<clip>.mp4 and refuses to start otherwise.

Pseudo-tactile add-on (pseudotactile/, for RGB LoRA stage 2 on the V4-formal architecture)

Extract the tars into the same HOT3D_EgoHOI root as the RGB package (members carry the full relative paths), then run localize_pseudotactile.py after make_local_manifest.py. The object 3D label archives (white_geometry/labels/<split>/<clip>.npz, ~150 GB) are provenance only and never read by training; they are not shipped and the localizer creates zero-filled placeholders of the recorded sizes so the loader's path/size checks pass. Full instructions: docs/RGB_LORA_STAGE2_V4FORMAL_PORTING.md in the code repository. The stage-1 checkpoint this version starts from is the model repo zhenyuxie-zhzh/wan22-egohoi-rgb-lora-stage1.

Pointmap add-on (pointmap/, for V6: RGB + combined hand/object pointmap MoT)

Each video encodes, for every visible hand/object surface pixel, its 3D position in the clip's first-frame camera frame: one fixed metric box for all clips, R=X, G=Y, B=Z, valid values 1..255, background exactly (0,0,0); decode with v = lower + (value - 1) / 254 * (upper - lower) (box in contract.json). Extract the three tars into the same HOT3D_EgoHOI root as the RGB package (members carry the full relative paths, so the product lands at <root>/hot3d_gt_pointmap_model_grade_v1/), then run localize_pointmap.py --root <root>/hot3d_gt_pointmap_model_grade_v1. Only the combined layer is shipped. The videos are H.264 in RGB mode; players without that support show a green background, ffmpeg-based decoders (what the loader uses) decode them correctly. Full instructions: docs/RGB_POINTMAP_LORA_V6_PORTING.md in the code repository. V6 needs no pretrained checkpoint.

Step-by-step porting guide

The repository document docs/RGB_OBJECT_ENTITY_PORTING.md (branch claude/rgb-lora-stage1) covers the environment (requirements-rgb-object-entity-nvidia.txt), model download, this dataset, training with 1-8 nodes (examples/sbatch_train_egohoi_hot3d_rgb_object_entity_nvidia.sh), evaluation and scoring (examples/run_inference_egohoi_val_rgb_object_entity_rgb74_nvidia.sh). docs/RGB_LORA_STAGE2_V4FORMAL_PORTING.md covers the RGB LoRA stage-2 (V4-formal) version, which also needs the pseudotactile/ add-on. docs/RGB_POINTMAP_LORA_V6_PORTING.md covers V6 (RGB + combined pointmap), which needs the pointmap/ add-on and no checkpoint.

How the code consumes it

Training (examples/train_egohoi_hot3d_rgb_object_entity.sh, wan22_base start):

export WAN_MODEL_DIR=<model_dir>                      # original Wan2.2-TI2V-5B + prompt_embeddings/
export HOT3D_TRAIN_DIR=/data/HOT3D_EgoHOI/train
export HOT3D_RGB_MANIFEST=/data/HOT3D_EgoHOI/hot3d_geometry_pseudotactile_v1/manifests/egohoi_rgb_manifest.jsonl
export RGB_OBJECT_ENTITY_START=wan22_base
# EGOHOI_NUM_PROCESSES x EGOHOI_GRADIENT_ACCUMULATION_STEPS must equal 64

Evaluation (examples/inference_egohoi_val_rgb_object_entity.py, RGB74 + oracle first-frame mask):

--dataset_base_path /data/HOT3D_EgoHOI/val \
--hot3d_rgb_manifest_path /data/HOT3D_EgoHOI/hot3d_geometry_pseudotactile_v1/manifests/egohoi_rgb_manifest.jsonl

The train loader fails closed unless all 1442 manifest clips have videos, skeleton videos, camera trajectories and contiguous 000000.png..000149.png masks; the evaluator requires exactly the 74 val clips.

Provenance

• —Source tree: /vast/users/zhenyu.xie/zhenyu/Datasets/HOT3D_EgoHOI on the MBZUAI cluster, packaged on 2026-09-17 (see MANIFEST.json for shard checksums and per-clip membership).
• —Manifest sha256 d469890b68a9bd3ca9fcd8bfac2b6bceaa7ce2a95b2c61c55d038513d0ac836b (egohoi_rgb_manifest.jsonl, 1516 rows, all official_split=train_aria).
• —Code: DiffSynth-Studio fork, branch claude/rgb-lora-stage1, object-entity files under diffsynth/*/wan_video_object_entity.py, diffsynth/core/data/hot3d_egohoi_object_entity.py, examples/*rgb_object_entity*.