Modelpublic

urikxx/EchoMimic

sourceHugging Faceupdated 2y agoView on Hugging Face

0likes6downloads

Model Card

<h1 align='center'>EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditioning</h1>

<div align='center'> <a href='https://github.com/yuange250' target='blank'>Zhiyuan Chen</a><sup>*</sup>&emsp; <a href='https://github.com/JoeFannie' target='blank'>Jiajiong Cao</a><sup></sup>&emsp; <a href='https://github.com/octavianChen' target='_blank'>Zhiquan Chen</a><sup></sup>&emsp; <a href='https://github.com/lymhust' target='_blank'>Yuming Li</a><sup></sup>&emsp; <a href='https://github.com/' target='_blank'>Chenguang Ma</a><sup></sup> </div> <div align='center'> Equal Contribution. </div>

<div align='center'> Terminal Technology Department, Alipay, Ant Group. </div> <br> <div align='center'> <a href='https://badtobest.github.io/echomimic.html'><img src='https://img.shields.io/badge/Project-Page-blue'></a> <a href='https://huggingface.co/BadToBest/EchoMimic'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace-Model-yellow'></a> <a href='https://huggingface.co/spaces/BadToBest/EchoMimic'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20HuggingFace-Demo-yellow'></a> <a href='https://www.modelscope.cn/models/BadToBest/EchoMimic'><img src='https://img.shields.io/badge/ModelScope-Model-purple'></a> <a href='https://www.modelscope.cn/studios/BadToBest/BadToBest'><img src='https://img.shields.io/badge/ModelScope-Demo-purple'></a> <a href='https://arxiv.org/abs/2407.08136'><img src='https://img.shields.io/badge/Paper-Arxiv-red'></a> <a href='assets/echomimic.png'><img src='https://badges.aleen42.com/src/wechat.svg'></a> </div>

📣 📣 Updates

—[2024.08.02] 🔥 EchoMimic is now available on huggingface with A100 GPU. Thanks Wenmeng Zhou@ModelScope.
—[2024.07.25] 🔥🔥🔥 Accelerated models and pipe on Audio Driven are released. The inference speed can be improved by 10x (from ~7mins/240frames to ~50s/240frames on V100 GPU)
—[2024.07.23] 🔥 EchoMimic gradio demo on modelscope is ready.
—[2024.07.23] 🔥 EchoMimic gradio demo on huggingface is ready. Thanks Sylvain Filoni@fffiloni.
—[2024.07.17] 🔥🔥🔥 Accelerated models and pipe on Audio + Selected Landmarks are released. The inference speed can be improved by 10x (from ~7mins/240frames to ~50s/240frames on V100 GPU)
—[2024.07.14] 🔥 ComfyUI is now available. Thanks @smthemex for the contribution.
—[2024.07.13] 🔥 Thanks NewGenAI for the video installation tutorial.
—[2024.07.13] 🔥 We release our pose&audio driven codes and models.
—[2024.07.12] 🔥 WebUI and GradioUI versions are released. We thank @greengerong @Robin021 and @O-O1024 for their contributions.
—[2024.07.12] 🔥 Our paper is in public on arxiv.
—[2024.07.09] 🔥 We release our audio driven codes and models.

Gallery

Audio Driven (Sing)

</table>

Audio Driven (English)

</table>

Audio Driven (Chinese)

</table>

Landmark Driven

</table>

Audio + Selected Landmark Driven

</table>

（Some demo images above are sourced from image websites. If there is any infringement, we will immediately remove them and apologize.）

Installation

Download the Codes

bash

  git clone https://github.com/BadToBest/EchoMimic
  cd EchoMimic

Python Environment Setup

—Tested System Environment: Centos 7.2/Ubuntu 22.04, Cuda >= 11.7
—Tested GPUs: A100(80G) / RTX4090D (24G) / V100(16G)
—Tested Python Version: 3.8 / 3.10 / 3.11

Create conda environment (Recommended):

bash

  conda create -n echomimic python=3.8
  conda activate echomimic

Install packages with pip

bash

  pip install -r requirements.txt

Download ffmpeg-static

Download and decompress ffmpeg-static, then

export FFMPEG_PATH=/path/to/ffmpeg-4.4-amd64-static

Download pretrained weights

shell

git lfs install
git clone https://huggingface.co/BadToBest/EchoMimic pretrained_weights

The pretrained_weights is organized as follows.

./pretrained_weights/
├── denoising_unet.pth
├── reference_unet.pth
├── motion_module.pth
├── face_locator.pth
├── sd-vae-ft-mse
│   └── ...
├── sd-image-variations-diffusers
│   └── ...
└── audio_processor
    └── whisper_tiny.pt

In which denoising_unet.pth / reference_unet.pth / motion_module.pth / face_locator.pth are the main checkpoints of EchoMimic. Other models in this hub can be also downloaded from it's original hub, thanks to their brilliant works:

Audio-Drived Algo Inference

Run the python inference script:

bash

  python -u infer_audio2vid.py
  python -u infer_audio2vid_pose.py

Audio-Drived Algo Inference On Your Own Cases

Edit the inference config file ./configs/prompts/animation.yaml, and add your own case:

bash

test_cases:
  "path/to/your/image":
    - "path/to/your/audio"

The run the python inference script:

bash

  python -u infer_audio2vid.py

Motion Alignment between Ref. Img. and Driven Vid.

(Firstly download the checkpoints with '_pose.pth' postfix from huggingface)

Edit drivervideo and refimage to your path in demomotionsync.py, then run

bash

  python -u demo_motion_sync.py

Audio&Pose-Drived Algo Inference

Edit ./configs/prompts/animation_pose.yaml, then run

bash

  python -u infer_audio2vid_pose.py

Pose-Drived Algo Inference

Set drawmouse=True in line 135 of inferaudio2vidpose.py. Edit ./configs/prompts/animationpose.yaml, then run

bash

  python -u infer_audio2vid_pose.py

Run the Gradio UI

Thanks to the contribution from @Robin021:

bash


python -u webgui.py --server_port=3000

Release Plans

Status	Milestone	ETA
✅	The inference source code of the Audio-Driven algo meet everyone on GitHub	9th July, 2024
✅	Pretrained models trained on English and Mandarin Chinese to be released	9th July, 2024
✅	The inference source code of the Pose-Driven algo meet everyone on GitHub	13th July, 2024
✅	Pretrained models with better pose control to be released	13th July, 2024
✅	Accelerated models to be released	17th July, 2024
🚀	Pretrained models with better sing performance to be released	TBD
🚀	Large-Scale and High-resolution Chinese-Based Talking Head Dataset	TBD

Acknowledgements

We would like to thank the contributors to the AnimateDiff, Moore-AnimateAnyone and MuseTalk repositories, for their open research and exploration.

We are also grateful to V-Express and hallo for their outstanding work in the area of diffusion-based talking heads.

If we missed any open-source projects or related articles, we would like to complement the acknowledgement of this specific work immediately.

Citation

If you find our work useful for your research, please consider citing the paper :

@misc{chen2024echomimic,
  title={EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditioning},
  author={Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, Chenguang Ma},
  year={2024},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}

Star History

![Star History Chart](https://star-history.com/?spm=5176.28103460.0.0.342a3da23STWrU#BadToBest/EchoMimic&Date)

&#x1F4E3; &#x1F4E3; Updates

Gallery

Audio Driven (Sing)

Audio Driven (English)

Audio Driven (Chinese)

Landmark Driven

Audio + Selected Landmark Driven

Installation

Download the Codes

Python Environment Setup

Download ffmpeg-static

Download pretrained weights

Audio-Drived Algo Inference

Audio-Drived Algo Inference On Your Own Cases

Motion Alignment between Ref. Img. and Driven Vid.

Audio&Pose-Drived Algo Inference

Pose-Drived Algo Inference

Run the Gradio UI

Release Plans

Acknowledgements

Citation

Star History

📣 📣 Updates