Priyam00/Video_Frame_Timestamp_Event_Localization

Text-Guided Temporal Action Localization Research Dashboard

This project implements a research-level prototype for Text-Guided Temporal Action Localization with Video Frame and Segment Timestamp Retrieval.

The user uploads a video and enters a natural-language query such as:

person falling down

The system extracts frames, matches the text query with frames using CLIP, predicts a relevant start-end timestamp segment, and displays research-ready metrics.

Major Features

• —Local video upload or YouTube video-link input
• —Frame extraction from uploaded/downloaded videos
• —CLIP / vision-language text-frame matching
• —Timestamp-based video segment retrieval
• —Statistical dashboard
• —Temporal IoU, Precision, Recall, F1-score, Boundary Error, Hit@0.5
• —Human positive / negative / corrected timestamp feedback
• —Frame-level feedback training labels
• —Feedback-trained score calibration model
• —Downloadable research logs and metric CSV files

YouTube Video Link Option

The dashboard now supports two video sources:

1. 1.Upload local video: upload mp4, avi, mov, mkv, or webm.
2. 2.YouTube video link: paste a public YouTube URL and the app downloads a temporary local copy for frame extraction.

Use YouTube videos only when you own the video, created it, or have permission to analyze it. Long videos can slow down Colab/CPU execution, so the sidebar includes a maximum YouTube duration limit.

Human Feedback Training

After each prediction, the reviewer can select:

1. 1.Correct localization / positive feedback
2. 2.Partially correct / corrected timestamps
3. 3.Wrong localization or misclassification / negative feedback

The system stores:

• —outputs/feedback/user_feedback.csv
• —outputs/feedback/frame_feedback_labels.csv
• —outputs/feedback/feedback_calibrator.joblib

The frame-level labels can be used for future supervised fine-tuning or for an ablation study in a research paper.

Important Note

This version trains a lightweight feedback calibration model over frame-level similarity features. It does not directly fine-tune the full CLIP model because CLIP fine-tuning requires a larger labeled dataset and GPU resources. The exported feedback labels can be used later for full deep model fine-tuning.

Run Locally

pip install -r requirements.txt
streamlit run app.py

Run in Google Colab

Upload the ZIP file, unzip it, install requirements, and run the commands given in COLAB_RUN_COMMANDS.md.

Research Paper Metrics

To calculate supervised metrics, enable:

I have true action start-end timestamps

Then enter the ground-truth start and end time. The dashboard will calculate:

• —Temporal IoU
• —Precision
• —Recall
• —F1-score
• —Boundary Error
• —Hit@0.5

Without ground truth, the dashboard can still show similarity statistics and predicted timestamp values, but it cannot calculate true supervised localization accuracy.