Jazzcharles/ego4d_train_pair_howto100m

📙 Overview

The metadata for Ego4d training set, with paired howto100m video clips. The ego-exo pair is constructed by choosing the ones with shared nouns/verbs.

Each sample represents a short video clip, which consists of

• —vid: the initial video id.
• —start_second: the start timestamp of the narration.
• —end_second: the end timestamp of the narration.
• —text: the original narration.
• —noun: a list containing the index of nouns in the Ego4d noun vocabulary.
• —verb: a list containing the index of verbs in the Ego4d verb vocabulary.
• —nvindex: a list of numbers indexing the [paired howto100m clips](https://huggingface.co/datasets/Jazzcharles/HowTo100Mllama3refinedcaption).

The noun and verb vocabularies can be found in EgoVLP (https://drive.google.com/drive/folders/16fUv5rrZmt06Ty3QAEweDpveC-84RI9Z).

🏋️ How-To-Use

Please refer to code EgoInstructor for details.

🎓 Citation

@article{xu2024retrieval,
  title={Retrieval-augmented egocentric video captioning},
  author={Xu, Jilan and Huang, Yifei and Hou, Junlin and Chen, Guo and Zhang, Yuejie and Feng, Rui and Xie, Weidi},
  journal={arXiv preprint arXiv:2401.00789},
  year={2024}
}