animatedaliensfans/Unsupervised-Generative-Video-Dubbing

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

Unsupervised Generative Video Dubbing

Author: Jimin Tan, Chenqin Yang, Yakun Wang, Yash Deshpande

Project Website: https://tanjimin.github.io/unsupervised-video-dubbing/

Training Code for the dubbing model is under the root directory. We used a pre-processed LRW for training. See data.py for details.

We created a simple depolyment pipeline which can be find under post_processing subdirectory. The pipeline takes the model weights we pre-trained on LRW. The pipeline takes a video and a equal duration audio segments and output a dubbed video based on audio information. See the instruction below for more details.

Requirement

• —LibROSA 0.7.2
• —dlib 19.19
• —OpenCV 4.2.0

• —Pillow 6.2.2
• —PyTorch 1.2.0
• —TorchVision 0.4.0

Post-Procesing Folder

.
├── source                  
│   ├── audio_driver_mp4    # contain audio drivers (saved in mp4 format)
│   ├── audio_driver_wav    # contain audio drivers (saved in wav format)
│   ├── base_video          # contain base videos (videos you'd like to modify)
│   ├── dlib            		# trained dlib models
│   └── model               # trained landmark generation models
├── main.py									# main function for post processing
├── main_support.py					# support functions used in main.py
├── models.py								# define the landmark generation model
├── step_3_vid2vid.sh		  	# Bash script for running vid2vid
├── step_4_denoise.sh.      # Bash script for denoising vid2vid results
├── compare_openness.ipynb  # mouth openness comparison across generated videos
└── README.md

- shapepredictor68facelandmarks.dat This is trained on the ibug 300-W dataset (https://ibug.doc.ic.ac.uk/resources/facial-point-annotations/) The license for this dataset excludes commercial use and Stefanos Zafeiriou, one of the creators of the dataset, asked me to include a note here saying that the trained model therefore can't be used in a commerical product. So you should contact a lawyer or talk to Imperial College London to find out if it's OK for you to use this model in a commercial product. {C. Sagonas, E. Antonakos, G, Tzimiropoulos, S. Zafeiriou, M. Pantic. 300 faces In-the-wild challenge: Database and results. Image and Vision Computing (IMAVIS), Special Issue on Facial Landmark Localisation "In-The-Wild". 2016.}

Detailed steps for model deployment

• —Go to post_processing directory
• —Run: ``python3 main.py -r step `` (corresponding step below)
• —e.g: python3 main.py -r 1 will run the first step and etc

Step 1 — generate landmarks

• —Input
• —Base video file path (./source/base_video/base_video.mp4)
• —Audio driver file path (./source/audio_driver_wav/audio_driver.wav)
• —Epoch (int)
• —Output (./result)
• —keypoints.npy (# generated landmarks in npy format)
• —source.txt (contains information about base video, audio driver, model epoch)
• —Process
• —Extract facial landmarks from base video
• —Extract MFCC features from driver audio
• —Pass MFCC features and facial landmarks into the model to retrieve mouth landmarks
• —Combine facial & mouth landmarks and save in npy format

Step 2 — Test generated frames

• —Input
• —None
• —Output (./result)
• —Folder — save_keypoints: visualized generated frames
• —Folder — savekeypointscsv : landmark coordinates for each frame, saved in txt format
• —openness.png: mouth openness measured and plotted across all frames
• —Process
• —Generate images from npy file
• —Generate openness plot

Step 3 — Execute vid2vid

• —Input
• —None
• —Output
• —Path for generated fake images from vid2vid are shown at the end; Please copy it back to the /result/vid2vid_frames/
• —Folder: vid2vid generated images
• —Process
• —Run vid2vid
• —Copy back vid2vid results to main folder

Step 4 — Denoise and smooth vid2vid results

• —Input
• —vid2vid generated images folder path
• —Original base images folder path
• —Output
• —Folder: Modified images (base image + vid2vid mouth regions)
• —Folder: Denoised and smoothed frames
• —Process
• —Crop mouth areas from vid2vid generated images and paste them back to base images —> modified image
• —Generate circular smoothed images by using gradient masking
• —Take (modified image, circular smoothed images) as pairs and do denoising

Step 5 — Generate modified videos with sound

• —Input
• —Saved frames folder path
• —By default, it is saved in ./result/save_keypoints; you can enter d to go with default path
• —Otherwise, input the frames folder path
• —Audio driver file path (./source/audio_driver_wav/audio_driver.wav)
• —Output (./result/save_keypoints/result/)
• —videowithoutsound.mp4: modified videos without sound
• —audio_only.mp4: audio driver
• —final_output.mp4: modified videos with sound
• —Process
• —Generate the modified video without sound with define fps
• —Extract wav from audio driver
• —Combine audio and video to generate final output

Important Notice

• —You may need to modify how MFCC features are extracted in extract_mfcc function
• —Be careful about sample rate, windowlength, hoplength
• —Good resource: https://www.mathworks.com/help/audio/ref/mfcc.html
• —You may need to modify the region of interest (mouth area) in frame_crop function
• —You may need to modify the frame rate defined in step_3 of the main.py, which should be your base video fps

# How to check your base video fps
# source: https://www.learnopencv.com/how-to-find-frame-rate-or-frames-per-second-fps-in-opencv-python-cpp/

import cv2
video = cv2.VideoCapture("video.mp4");

# Find OpenCV version
(major_ver, minor_ver, subminor_ver) = (cv2.__version__).split('.')
if int(major_ver)  < 3 :
    fps = video.get(cv2.cv.CV_CAP_PROP_FPS)
    print("Frames per second using video.get(cv2.cv.CV_CAP_PROP_FPS): {0}".format(fps))
else :
    fps = video.get(cv2.CAP_PROP_FPS)
    print("Frames per second using video.get(cv2.CAP_PROP_FPS) : {0}".format(fps))
video.release()

• —You may need to modify the shell path

echo $SHELL

• —You may need to modify the audio sampling rate in extract_audio function
• —You may need to customize your parameters in combine_audio_video function
• —Good resource: https://ffmpeg.org/ffmpeg.html
• —https://gist.github.com/tayvano/6e2d456a9897f55025e25035478a3a50

Update History

• —March 22, 2020: Drafted documentation