oznr126/echomimic-v2

1<h1 align='center'>EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animation</h1>
2
3<div align='center'>
4    <a href='https://github.com/mengrang' target='_blank'>Rang Meng</a><sup></sup>&emsp;
5    <a href='https://github.com/' target='_blank'>Xingyu Zhang</a><sup></sup>&emsp;
6    <a href='https://lymhust.github.io/' target='_blank'>Yuming Li</a><sup></sup>&emsp;
7    <a href='https://github.com/' target='_blank'>Chenguang Ma</a><sup></sup>
8</div>
9
10
11<div align='center'>
12Terminal Technology Department, Alipay, Ant Group.
13</div>
14<br>
15<div align='center'>
16    <a href='https://antgroup.github.io/ai/echomimic_v2/'><img src='https://img.shields.io/badge/Project-Page-blue'></a>
17    <a href='https://huggingface.co/BadToBest/EchoMimicV2'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace-Model-yellow'></a>
18    <!--<a href='https://antgroup.github.io/ai/echomimic_v2/'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace-Demo-yellow'></a>-->
19    <a href='https://modelscope.cn/models/BadToBest/EchoMimicV2'><img src='https://img.shields.io/badge/ModelScope-Model-purple'></a>
20    <!--<a href='https://antgroup.github.io/ai/echomimic_v2/'><img src='https://img.shields.io/badge/ModelScope-Demo-purple'></a>-->
21    <a href='https://arxiv.org/abs/2411.10061'><img src='https://img.shields.io/badge/Paper-Arxiv-red'></a>
22    <a href='https://github.com/antgroup/echomimic_v2/blob/main/assets/halfbody_demo/wechat_group.png'><img src='https://badges.aleen42.com/src/wechat.svg'></a>
23</div>
24<div align='center'>
25    <a href='https://github.com/antgroup/echomimic_v2/discussions/53'><img src='https://img.shields.io/badge/English-Common Problems-orange'></a>
26    <a href='https://github.com/antgroup/echomimic_v2/discussions/40'><img src='https://img.shields.io/badge/中文版-常见问题汇总-orange'></a>
27</div>
28
29## &#x1F680; EchoMimic Series
30* EchoMimicV1: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditioning. [GitHub](https://github.com/antgroup/echomimic)
31* EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animation. [GitHub](https://github.com/antgroup/echomimic_v2)
32
33## &#x1F4E3; Updates
34* [2024.11.27] 🔥 Thanks [AiMotionStudio](https://www.youtube.com/@AiMotionStudio) for the [installation tutorial](https://www.youtube.com/watch?v=2ab6U1-nVTQ).
35* [2024.11.22] 🔥 [GradioUI](https://github.com/antgroup/echomimic_v2/blob/main/app.py) is now available. Thanks @gluttony-10 for the contribution.
36* [2024.11.22] 🔥 [ComfyUI](https://github.com/smthemex/ComfyUI_EchoMimic) is now available. Thanks @smthemex for the contribution.
37* [2024.11.21] 🔥 We release the EMTD dataset list and processing scripts.
38* [2024.11.21] 🔥 We release our [EchoMimicV2](https://github.com/antgroup/echomimic_v2) codes and models.
39* [2024.11.15] 🔥 Our [paper](https://arxiv.org/abs/2411.10061) is in public on arxiv.
40
41## &#x1F305; Gallery
42### Introduction
43<table class="center">
44<tr>
45    <td width=50% style="border: none">
46        <video controls loop src="https://github.com/user-attachments/assets/f544dfc0-7d1a-4c2c-83c0-608f28ffda25" muted="false"></video>
47    </td>
48    <td width=50% style="border: none">
49        <video controls loop src="https://github.com/user-attachments/assets/7f626b65-725c-4158-a96b-062539874c63" muted="false"></video>
50    </td>
51</tr>
52</table>
53
54### English Driven Audio
55<table class="center">
56<tr>
57    <td width=100% style="border: none">
58        <video controls loop src="https://github.com/user-attachments/assets/3d5ac52c-62e4-41bc-8b27-96f005bbd781" muted="false"></video>
59    </td>
60</tr>
61</table>
62<table class="center">
63<tr>
64    <td width=30% style="border: none">
65        <video controls loop src="https://github.com/user-attachments/assets/e8dd6919-665e-4343-931f-54c93dc49a7d" muted="false"></video>
66    </td>
67    <td width=30% style="border: none">
68        <video controls loop src="https://github.com/user-attachments/assets/2a377391-a0d3-4a9d-8dde-cc59006e7e5b" muted="false"></video>
69    </td>
70    <td width=30% style="border: none">
71        <video controls loop src="https://github.com/user-attachments/assets/462bf3bb-0af2-43e2-a2dc-559e79953f3c" muted="false"></video>
72    </td>
73</tr>
74<tr>
75    <td width=30% style="border: none">
76        <video controls loop src="https://github.com/user-attachments/assets/0e988e7f-6346-4b54-9061-9cfc7a80e9c8" muted="false"></video>
77    </td>
78    <td width=30% style="border: none">
79        <video controls loop src="https://github.com/user-attachments/assets/56f739bd-afbf-4ed3-ab15-73a811c1bc46" muted="false"></video>
80    </td>
81    <td width=30% style="border: none">
82        <video controls loop src="https://github.com/user-attachments/assets/1b2f7827-111d-4fc0-a773-e1731bba285d" muted="false"></video>
83    </td>
84</tr>
85<tr>
86    <td width=30% style="border: none">
87        <video controls loop src="https://github.com/user-attachments/assets/a76b6cc8-89b9-4f7e-b1ce-c85a657b6dc7" muted="false"></video>
88    </td>
89    <td width=30% style="border: none">
90        <video controls loop src="https://github.com/user-attachments/assets/bf03b407-5033-4a30-aa59-b8680a515181" muted="false"></video>
91    </td>
92    <td width=30% style="border: none">
93        <video controls loop src="https://github.com/user-attachments/assets/f98b3985-572c-499f-ae1a-1b9befe3086f" muted="false"></video>
94    </td>
95</tr>
96</table>
97
98### Chinese Driven Audio
99<table class="center">
100<tr>
101    <td width=30% style="border: none">
102        <video controls loop src="https://github.com/user-attachments/assets/a940a332-2fd1-48e7-b3c4-f88f63fd1c9d" muted="false"></video>
103    </td>
104    <td width=30% style="border: none">
105        <video controls loop src="https://github.com/user-attachments/assets/8f185829-c67f-45f4-846c-fcbe012c3acf" muted="false"></video>
106    </td>
107    <td width=30% style="border: none">
108        <video controls loop src="https://github.com/user-attachments/assets/a49ab9be-f17b-41c5-96dd-20dc8d759b45" muted="false"></video>
109    </td>
110</tr>
111<tr>
112    <td width=30% style="border: none">
113        <video controls loop src="https://github.com/user-attachments/assets/1136ec68-a13c-4ee7-ab31-5621530bf9df" muted="false"></video>
114    </td>
115    <td width=30% style="border: none">
116        <video controls loop src="https://github.com/user-attachments/assets/fc16d512-8806-4662-ae07-8fcf45c75a83" muted="false"></video>
117    </td>
118    <td width=30% style="border: none">
119        <video controls loop src="https://github.com/user-attachments/assets/f8559cd1-f555-4781-9251-dfcef10b5b01" muted="false"></video>
120    </td>
121</tr>
122<tr>
123    <td width=30% style="border: none">
124        <video controls loop src="https://github.com/user-attachments/assets/c7473e3a-ab51-4ad5-be96-6c4691fc0c6e" muted="false"></video>
125    </td>
126    <td width=30% style="border: none">
127        <video controls loop src="https://github.com/user-attachments/assets/ca69eac0-5126-41ee-8cac-c9722004d771" muted="false"></video>
128    </td>
129    <td width=30% style="border: none">
130        <video controls loop src="https://github.com/user-attachments/assets/e66f1712-b66d-46b5-8bbd-811fbcfea4fd" muted="false"></video>
131    </td>
132</tr>
133</table>
134
135## ⚒️ Installation
136### Download the Codes
137
138```bash
139  git clone https://github.com/antgroup/echomimic_v2
140  cd echomimic_v2
141```
142
143### Python Environment Setup
144
145- Tested System Environment: Centos 7.2/Ubuntu 22.04, Cuda >= 11.7
146- Tested GPUs: A100(80G) / RTX4090D (24G) / V100(16G)
147- Tested Python Version: 3.8 / 3.10 / 3.11
148
149Create conda environment (Recommended):
150
151```bash
152  conda create -n echomimic python=3.10
153  conda activate echomimic
154```
155
156Install packages with `pip`
157```bash
158  pip install pip -U
159  pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 xformers==0.0.28.post3 --index-url https://download.pytorch.org/whl/cu124
160  pip install torchao --index-url https://download.pytorch.org/whl/nightly/cu124
161  pip install -r requirements.txt
162  pip install --no-deps facenet_pytorch==2.6.0
163```
164
165### Download ffmpeg-static
166Download and decompress [ffmpeg-static](https://www.johnvansickle.com/ffmpeg/old-releases/ffmpeg-4.4-amd64-static.tar.xz), then
167```
168export FFMPEG_PATH=/path/to/ffmpeg-4.4-amd64-static
169```
170
171### Download pretrained weights
172
173```shell
174git lfs install
175git clone https://huggingface.co/BadToBest/EchoMimicV2 pretrained_weights
176```
177
178The **pretrained_weights** is organized as follows.
179
180```
181./pretrained_weights/
182├── denoising_unet.pth
183├── reference_unet.pth
184├── motion_module.pth
185├── pose_encoder.pth
186├── sd-vae-ft-mse
187│   └── ...
188├── sd-image-variations-diffusers
189│   └── ...
190└── audio_processor
191    └── tiny.pt
192```
193
194In which **denoising_unet.pth** / **reference_unet.pth** / **motion_module.pth** / **pose_encoder.pth** are the main checkpoints of **EchoMimic**. Other models in this hub can be also downloaded from it's original hub, thanks to their brilliant works:
195- [sd-vae-ft-mse](https://huggingface.co/stabilityai/sd-vae-ft-mse)
196- [sd-image-variations-diffusers](https://huggingface.co/lambdalabs/sd-image-variations-diffusers)
197- [audio_processor(whisper)](https://openaipublic.azureedge.net/main/whisper/models/65147644a518d12f04e32d6f3b26facc3f8dd46e5390956a9424a650c0ce22b9/tiny.pt)
198
199### Inference on Demo 
200Run the gradio:
201```bash
202python app.py
203```
204Run the python inference script:
205```bash
206python infer.py --config='./configs/prompts/infer.yaml'
207```
208
209### EMTD Dataset
210Download dataset:
211```bash
212python ./EMTD_dataset/download.py
213```
214Slice dataset:
215```bash
216bash ./EMTD_dataset/slice.sh
217```
218Process dataset:
219```bash
220python ./EMTD_dataset/preprocess.py
221```
222
223## 📝 Release Plans
224
225|  Status  | Milestone                                                                | ETA |
226|:--------:|:-------------------------------------------------------------------------|:--:|
227|    ✅    | The inference source code of EchoMimicV2 meet everyone on GitHub   | 21st Nov, 2024 |
228|    ✅    | Pretrained models trained on English and Mandarin Chinese on HuggingFace | 21st Nov, 2024 |
229|    ✅    | Pretrained models trained on English and Mandarin Chinese on ModelScope   | 21st Nov, 2024 |
230|    ✅    | EMTD dataset list and processing scripts                | 21st Nov, 2024 |
231|    🚀    | Accelerated models to be released                                        | TBD |
232|    🚀    | Online Demo on ModelScope to be released            | TBD |
233|    🚀    | Online Demo on HuggingFace to be released         | TBD |
234
235## ⚖️ Disclaimer
236This project is intended for academic research, and we explicitly disclaim any responsibility for user-generated content. Users are solely liable for their actions while using the generative model. The project contributors have no legal affiliation with, nor accountability for, users' behaviors. It is imperative to use the generative model responsibly, adhering to both ethical and legal standards.
237
238## 🙏🏻 Acknowledgements
239
240We would like to thank the contributors to the [MimicMotion](https://github.com/Tencent/MimicMotion) and [Moore-AnimateAnyone](https://github.com/MooreThreads/Moore-AnimateAnyone) repositories, for their open research and exploration. 
241
242We are also grateful to [CyberHost](https://cyberhost.github.io/) and [Vlogger](https://enriccorona.github.io/vlogger/) for their outstanding work in the area of audio-driven human animation.
243
244If we missed any open-source projects or related articles, we would like to complement the acknowledgement of this specific work immediately.
245
246## &#x1F4D2; Citation
247
248If you find our work useful for your research, please consider citing the paper :
249
250```
251@misc{meng2024echomimic,
252  title={EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animation},
253  author={Rang Meng, Xingyu Zhang, Yuming Li, Chenguang Ma},
254  year={2024},
255  eprint={2411.10061},
256  archivePrefix={arXiv},
257  primaryClass={cs.CV}
258}
259```
260
261## &#x1F31F; Star History
262[![Star History Chart](https://api.star-history.com/svg?repos=antgroup/echomimic_v2&type=Date)](https://star-history.com/#antgroup/echomimic_v2&Date)
263