CoolFace
Apppublic

Agents-MCP-Hackathon/SayujM_Video_Person_Analysis

sourceHugging Faceupdated 1y agoView on Hugging Face
0likes
App README

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

Faces in Frames: AI-Powered Video Person Analysis (MCP Server)

Hackathon Track: mcp-server-track

Project Description

This Gradio application functions as an MCP Server designed for comprehensive person analysis within video content. It offers a robust pipeline for identifying, tracking, and clustering human faces, providing valuable insights into video demographics and interactions.

Core Functionality:

  • —Video Frame Extraction: Accepts a video file and extracts individual frames at a user-specified rate (frames per second).
  • —Human Face Detection: Utilizes state-of-the-art YOLO and YOLO-Face models to accurately identify bounding boxes for human faces within the extracted .jpg frames.
  • —Face Embedding Generation: Leverages the powerful DeepFace model (specifically ArcFace) to generate unique numerical embeddings for each detected human face.
  • —Temporal Aggregation: Employs temporal aggregation techniques to combine various face embeddings obtained across different frames, enhancing consistency and robustness.
  • —Agglomerative Clustering: Performs agglomerative clustering on the aggregated face embeddings, grouping similar faces based on a user-defined distance threshold. This helps in identifying unique individuals within the video.
  • —Distance Distribution Statistics: Provides essential statistics (minimum, maximum, mean, median) for the pairwise distance distribution of the face embeddings. This allows users to fine-tune their input clustering distance threshold for optimal results.
  • —HTML Report Generation: Generates a comprehensive HTML report detailing the analysis results, including clustered faces, timestamps, and other relevant information.

How to Use the Gradio Application

  1. 1.Upload your video file: Drag and drop or browse to select the video you wish to analyze.
  2. 2.Adjust "Frames per second to extract" slider: Control the density of frames processed from your video. A higher rate will capture more detail but may increase processing time.
  3. 3.Set an initial value for "Face Clustering Distance Threshold": This parameter dictates how close two face embeddings must be to be considered the same person. Start with an initial value, which you can refine later.
  4. 4.Click "Run Analysis": Initiate the video processing and face analysis.
  5. 5.Review Statistics and HTML Report: Once the initial analysis is complete, examine the generated HTML report and the provided statistics (min, max, mean, median pairwise distance for face embeddings). These statistics are crucial for understanding the distribution of face similarities and fine-tuning your clustering.
  6. 6.Fine-tune Threshold (Optional): Based on the statistics and the visual results in the HTML report, adjust the "Face Clustering Distance Threshold" to achieve more accurate clustering of individuals.
  7. 7.Click "Rerun Analysis" (Optional): If you've adjusted the threshold, click this button to re-run the clustering with the new parameter.
  8. 8.Detailed HTML Report: The updated detailed HTML report will be displayed, reflecting the changes from your refined threshold.

MCP Server in Action

This Gradio Space also functions as an MCP (Multi-Modal Chat Protocol) server. Below is a link to a video demonstrating this MCP server in action, interacting with an MCP client. This showcases the server's ability to receive and process requests from compatible clients.

MCP Server Demo Video: [Link to be added as an MCP Client is built]

Project Repository

For a deeper dive into the code and project structure, please visit the GitHub repository:

GitHub Repository: [Link to be added after creating a public repo]