anonseoul/turn-taking-dataset

turn-taking-detection-dataset

scripts/notebooks for turn taking detection dataset

Frame Feature extraction

Currently, extracting feature with mmaciton: current method needs to be fix. It is difference from the OAD convention. current method: please refer this Only taking the central frame among non-overlapping video snippets. In OAD task, all the frames in from snippets are converted into the mean of features from them.

To-dos (legacy, now maintain @ notion)

Data Preprocessing

• —[x] Split ego4d videos
• —[x] Extract RGB rawframes (ongoing) We do not use clip-level feature, we need to extract all rawframes from each video, for detail, see LSTR repo below
• —[ ] (secondary) Extract flow rawframes (getting flow rawframes is tricky because of installing denseflow)
• —[ ] Convert transcript timestamp into frame-level annotation Still do not know how to convert frame-level annoation of a video into frame-level annotation of a feature

Feature extraction (this would be help for extracting features)

• —[ ] Extract RGB features with TSN, following instructions
• —[ ] Extract audio features
• —[ ] (secondary) Extract flow features

Problems

Current extracted feature does not fit to the testra feature (Ours temporal dimension T in [T x 2048] are rounded, while testra feature are always rounded down)

Docs

Following LSTR, planning to use mmaction 0.x instead of 1.x, be careful with the mmaction version LSTR also followed mmaction2 activitynet pipeline, and convert features into .npy Many details are in LSTR and mmaction issues

mmaction dataset preparation doc mmaction feature extraction (based on 0.x, deprecated) mmaction preparing AcitivityNet mmaction preparing THUMOS14 mmaction TSN