ut-vision/EgoHaFL

EgoHaFL: Egocentric 3D Hand Forecasting Dataset with Language Instruction

EgoHaFL is a dataset designed for egocentric (first-person) 3D hand forecasting with accompanying natural language instructions. It was introduced in the paper SFHand: Learning Embodied Manipulation by Streaming Egocentric 3D Hand Forecasting.

The dataset contains short video clips, text descriptions, camera intrinsics, and detailed MANO-based 3D hand annotations. The dataset supports research in 3D hand forecasting, hand pose estimation, hand–object interaction understanding, and video–language modeling.

[image]

  

📦 Dataset Contents

1. Metadata CSV Files

• —EgoHaFL_train.csv
• —EgoHaFL_test.csv

Each row corresponds to one sample and contains:

2. 3D Hand Annotations (EgoHaFL_lmdb)

The folder EgoHaFL_lmdb stores all 3D annotations in LMDB format.

• —Key: uid
• —Value: a list of length 16, representing uniformly sampled frames across a 3-second video segment.

Each of the 16 elements is a dictionary containing:

• —mano_params

• —global_orient (n, 1, 3 ,3)
• —hand_pose (n, 15, 3, 3)
• —betas (n, 10)
• —is_right (n,)
• —keypoints_3d (n, 21, 3)
• —keypoints_2d (n, 21, 2)
• —vertices (n, 778, 3)
• —box_center (n, 2)
• —box_size (n,)
• —camera_t (n, 3) 3D hand position in camera coordinate
• —focal_length (n, 2)

Here, n denotes the number of hands present in each frame, which may vary across frames. When no hands are detected, the dictionary is empty.

🌳 Annotation Structure (Tree View)

Below is the hierarchical structure for a single annotation entry (uid → 16-frame list → per-frame dict):

<uid>
└── list (length = 16)
    ├── [0]
    │   ├── mano_params
    │   │   ├── global_orient
    │   │   ├── hand_pose
    │   │   └── betas
    │   ├── is_right
    │   ├── keypoints_3d
    │   ├── keypoints_2d
    │   ├── vertices
    │   ├── box_center
    │   ├── box_size
    │   ├── camera_t
    │   └── focal_length
    ├── [1]
    │   └── ...
    ├── [2]
    │   └── ...
    └── ...

🎥 Source of Video Data

The video clips used in EgoHaFL originate from the Ego4D V1 dataset. For our experiments, we use the original-length videos compressed to 224p resolution to ensure efficient storage and training.

Official Ego4D website: 🔗 [https://ego4d-data.org/](https://ego4d-data.org/)

🧩 Example of Use

For details on how to load and use the EgoHaFL dataset, please refer to the dataloader implementation in our GitHub repository:

🔗 [https://github.com/ut-vision/SFHand](https://github.com/ut-vision/SFHand)

🧠 Supported Research Tasks

• —Egocentric 3D hand forecasting
• —Hand motion prediction and trajectory modeling
• —3D hand pose estimation
• —Hand–object interaction understanding
• —Video–language multimodal modeling
• —Temporal reasoning with 3D human hands

📚 Citation

If you use this dataset or find it helpful in your research, please cite:

@article{liu2025sfhand,
  title={SFHand: A Streaming Framework for Language-guided 3D Hand Forecasting and Embodied Manipulation},
  author={Liu, Ruicong and Huang, Yifei and Ouyang, Liangyang and Kang, Caixin and Sato, Yoichi},
  journal={arXiv preprint arXiv:2511.18127},
  year={2025}
}