CoolFace
Apppublic

Ritikumar/real_time_face_segmentation

sourceHugging Faceupdated 6mo agoView on Hugging Face
0likes
App README

Real-Time Face Segmentation for Movie Cast

A deep learning project that detects and segments faces from movie scene screenshots and displays the predicted face region using a binary mask and bounding box through a Streamlit web app.

Project Identification

  • —Project Title: Real-Time Face Segmentation for Movie Cast
  • —Domain: Entertainment
  • —Skills Takeaway: Python, Computer Vision, OpenCV, Streamlit

Problem Statement

Scene Cast AI is designed for a streaming platform where users can pause a movie scene and instantly view cast or crew details for actors visible on screen.

The objective of this project is to automatically detect and segment face regions from movie scene screenshots so that the system can support cast identification and related user features.

Business Domain Value

  • —Pause-and-identify feature for actor name and profile overlays.
  • —Personalized recommendations based on user interest in specific actors.
  • —Support for content moderation through face-related visual analysis.
  • —Advertising opportunities using actor-based targeting during streaming breaks.

Project Overview

This project focuses on face segmentation, where the model predicts the face region at pixel level instead of only classifying the image.

A U-Net segmentation architecture with a MobileNetV2 encoder is used to generate face masks from movie scene images. The trained model is then integrated into a Streamlit application where users can upload an image and view the prediction results.

Skills and Tools Used

  • —Python
  • —NumPy
  • —OpenCV
  • —TensorFlow / Keras
  • —Matplotlib
  • —Seaborn
  • —Streamlit
  • —PIL
  • —Deep Learning
  • —Computer Vision

Dataset

The dataset contains movie scene images and corresponding face annotations / masks.

Dataset Link: Google Drive Dataset

Project Approach

1. Preprocessing, Data Visualization, and EDA

  • —Loaded movie scene images and corresponding annotation data.
  • —Converted annotations into binary masks.
  • —Resized images and masks to 256 x 256.
  • —Applied preprocessing compatible with MobileNetV2.
  • —Visualized sample images and masks for inspection.

2. Model Building

  • —Built a U-Net model with MobileNetV2 encoder using transfer learning.
  • —Added custom decoder blocks with skip connections.
  • —Trained the segmentation model on processed movie scene data.
  • —Saved model weights for reuse without retraining from scratch.

3. Testing and Fine-Tuning

  • —Generated predictions on validation images.
  • —Applied thresholding to convert predicted masks into binary masks.
  • —Evaluated the model using segmentation metrics.
  • —Improved output visualization using bounding boxes in the Streamlit app.

4. Deployment

  • —Integrated the trained model into a Streamlit web application.
  • —Added image upload functionality for prediction.
  • —Displayed original image, predicted mask, and bounding box output.

Model Architecture

This project uses:

  • —Encoder: MobileNetV2
  • —Decoder: U-Net style decoder with upsampling and skip connections
  • —Input Shape: 256 x 256 x 3
  • —Output Shape: 256 x 256 x 1 segmentation mask

Streamlit App Features

  • —Upload image for prediction
  • —Automatic preprocessing
  • —Predicted binary face mask generation
  • —Bounding box visualization on original image
  • —User-friendly interactive interface

Evaluation Metrics

Best model manual evaluation metrics:

  • —Dice Coefficient: 0.6193
  • —IoU: 0.4682
  • —Precision: 0.8359
  • —Recall: 0.5823
  • —F1-Score: 0.6193

Result Interpretation

  • —The model shows strong precision, which means predicted face regions are often correct when detected.
  • —Recall is lower than precision, which suggests the model misses some parts of the face region in certain images.
  • —Dice and IoU indicate that the current model works as a baseline segmentation system and can be improved further with more tuning and training.
  • —The current implementation is suitable for demonstrating the complete deep learning pipeline from preprocessing to deployment.

Output

The Streamlit application currently provides:

  • —Original uploaded image
  • —Predicted binary mask
  • —Bounding box result on detected face region

Folder Structure

bash
project-folder/
│── app.py
│── README.md
│── requirements.txt
│── savedmodels/
│   └── unet_mobilenetv2_best.weights.h5
│──  Copy_of_Real_Time_Face_Segmentation.ipynb

How to Run the Project

1. Clone the repository

bash
git clone <your-github-repository-link>
cd <your-project-folder>

2. Install dependencies

bash
pip install -r requirements.txt

3. Run the Streamlit application

bash
streamlit run app.py

Project Deliverables

  • —Jupyter Notebook with EDA, preprocessing, training, and evaluation
  • —Trained U-Net model weights (.h5)
  • —Streamlit application
  • —README.md
  • —requirements.txt
  • —Sample prediction outputs
  • —Demo video

Future Improvements

  • —Add webcam input for real-time prediction
  • —Add downloadable output images
  • —Add performance dashboard in Streamlit
  • —Improve segmentation with better augmentation and hyperparameter tuning
  • —Extend project toward actor identification after segmentation

Technical Tags

  • —Python
  • —OpenCV
  • —Streamlit
  • —TensorFlow
  • —Deep Learning
  • —Computer Vision
  • —Image Segmentation
  • —U-Net
  • —MobileNetV2

Author

Ritik

License

This project is created for learning, academic, and portfolio purposes.