faraidun/LivePortrait

1---2license: mit3---4 5<h1 align="center">LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control</h1>6 7<div align='center'>8    <a href='https://github.com/cleardusk' target='_blank'><strong>Jianzhu Guo</strong></a><sup> 1†</sup>&emsp;9    <a href='https://github.com/KwaiVGI' target='_blank'><strong>Dingyun Zhang</strong></a><sup> 1,2</sup>&emsp;10    <a href='https://github.com/KwaiVGI' target='_blank'><strong>Xiaoqiang Liu</strong></a><sup> 1</sup>&emsp;11    <a href='https://github.com/KwaiVGI' target='_blank'><strong>Zhizhou Zhong</strong></a><sup> 1,3</sup>&emsp;12    <a href='https://scholar.google.com.hk/citations?user=_8k1ubAAAAAJ' target='_blank'><strong>Yuan Zhang</strong></a><sup> 1</sup>&emsp;13</div>14 15<div align='center'>16    <a href='https://scholar.google.com/citations?user=P6MraaYAAAAJ' target='_blank'><strong>Pengfei Wan</strong></a><sup> 1</sup>&emsp;17    <a href='https://openreview.net/profile?id=~Di_ZHANG3' target='_blank'><strong>Di Zhang</strong></a><sup> 1</sup>&emsp;18</div>19 20<div align='center'>21    <sup>1 </sup>Kuaishou Technology&emsp; <sup>2 </sup>University of Science and Technology of China&emsp; <sup>3 </sup>Fudan University&emsp;22</div>23 24<br>25<div align="center" style="display: flex; justify-content: center; flex-wrap: wrap;">26  <!-- <a href='LICENSE'><img src='https://img.shields.io/badge/license-MIT-yellow'></a> -->27  <a href='https://arxiv.org/pdf/2407.03168'><img src='https://img.shields.io/badge/arXiv-LivePortrait-red'></a>28  <a href='https://liveportrait.github.io'><img src='https://img.shields.io/badge/Project-LivePortrait-green'></a>29  <a href='https://huggingface.co/spaces/KwaiVGI/liveportrait'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue'></a>30</div>31<br>32 33<p align="center">34  <img src="./docs/showcase2.gif" alt="showcase">35  <br>36  🔥 For more results, visit our <a href="https://liveportrait.github.io/"><strong>homepage</strong></a> 🔥37</p>38 39 40 41## 🔥 Updates42- **`2024/07/04`**: 🔥 We released the initial version of the inference code and models. Continuous updates, stay tuned!43- **`2024/07/04`**: 😊 We released the [homepage](https://liveportrait.github.io) and technical report on [arXiv](https://arxiv.org/pdf/2407.03168).44 45## Introduction46This repo, named **LivePortrait**, contains the official PyTorch implementation of our paper [LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control](https://arxiv.org/pdf/2407.03168).47We are actively updating and improving this repository. If you find any bugs or have suggestions, welcome to raise issues or submit pull requests (PR) 💖.48 49## 🔥 Getting Started50### 1. Clone the code and prepare the environment51```bash52git clone https://github.com/KwaiVGI/LivePortrait53cd LivePortrait54 55# create env using conda56conda create -n LivePortrait python==3.9.1857conda activate LivePortrait58# install dependencies with pip59pip install -r requirements.txt60```61 62### 2. Download pretrained weights63Download our pretrained LivePortrait weights and face detection models of InsightFace from [Google Drive](https://drive.google.com/drive/folders/1UtKgzKjFAOmZkhNK-OYT0caJ_w2XAnib) or [Baidu Yun](https://pan.baidu.com/s/1MGctWmNla_vZxDbEp2Dtzw?pwd=z5cn). We have packed all weights in one directory 😊. Unzip and place them in `./pretrained_weights` ensuring the directory structure is as follows:64```text65pretrained_weights66├── insightface67│   └── models68│       └── buffalo_l69│           ├── 2d106det.onnx70│           └── det_10g.onnx71└── liveportrait72    ├── base_models73    │   ├── appearance_feature_extractor.pth74    │   ├── motion_extractor.pth75    │   ├── spade_generator.pth76    │   └── warping_module.pth77    ├── landmark.onnx78    └── retargeting_models79        └── stitching_retargeting_module.pth80```81 82### 3. Inference 🚀83 84```bash85python inference.py86```87 88If the script runs successfully, you will get an output mp4 file named `animations/s6--d0_concat.mp4`. This file includes the following results: driving video, input image, and generated result.89 90<p align="center">91  <img src="./docs/inference.gif" alt="image">92</p>93 94Or, you can change the input by specifying the `-s` and `-d` arguments:95 96```bash97python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d0.mp498 99# or disable pasting back100python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d0.mp4 --no_flag_pasteback101 102# more options to see103python inference.py -h104```105 106**More interesting results can be found in our [Homepage](https://liveportrait.github.io)** 😊107 108### 4. Gradio interface109 110We also provide a Gradio interface for a better experience, just run by:111 112```bash113python app.py114```115 116### 5. Inference speed evaluation 🚀🚀🚀117We have also provided a script to evaluate the inference speed of each module:118 119```bash120python speed.py121```122 123Below are the results of inferring one frame on an RTX 4090 GPU using the native PyTorch framework with `torch.compile`:124 125| Model                             | Parameters(M) | Model Size(MB) | Inference(ms) |126|-----------------------------------|:-------------:|:--------------:|:-------------:|127| Appearance Feature Extractor      |     0.84      |       3.3      |     0.82      |128| Motion Extractor                  |     28.12     |       108      |     0.84      |129| Spade Generator                   |     55.37     |       212      |     7.59      |130| Warping Module                    |     45.53     |       174      |     5.21      |131| Stitching and Retargeting Modules|     0.23      |       2.3      |     0.31      |132 133*Note: the listed values of Stitching and Retargeting Modules represent the combined parameter counts and the total sequential inference time of three MLP networks.*134 135 136## Acknowledgements137We would like to thank the contributors of [FOMM](https://github.com/AliaksandrSiarohin/first-order-model), [Open Facevid2vid](https://github.com/zhanglonghao1992/One-Shot_Free-View_Neural_Talking_Head_Synthesis), [SPADE](https://github.com/NVlabs/SPADE), [InsightFace](https://github.com/deepinsight/insightface) repositories, for their open research and contributions.138 139## Citation 💖140If you find LivePortrait useful for your research, welcome to 🌟 this repo and cite our work using the following BibTeX:141```bibtex142@article{guo2024live,143  title   = {LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control},144  author  = {Jianzhu Guo and Dingyun Zhang and Xiaoqiang Liu and Zhizhou Zhong and Yuan Zhang and Pengfei Wan and Di Zhang},145  year    = {2024},146  journal = {arXiv preprint:2407.03168},147}148```149