AXERA-TECH/LivePortrait

1---2license: mit3language:4- en5base_model:6- KwaiVGI/LivePortrait7pipeline_tag: image-to-video8---9 10<p align="center">11  <img src="./assets/showcase2.gif" alt="showcase">12</p>13 14# LivePortrait15 16This version of LivePortrait has been converted to run on the Axera NPU using **w8a16** quantization.17 18This model has been optimized with the following:19 20Compatible with Pulsar2 version: 3.421 22## Convert tools links:23 24For those who are interested in model conversion:25 26- [the original repo](https://huggingface.co/KwaiVGI/LivePortrait)27- [Github for LivePortrait](https://github.com/AXERA-TECH/LivePortrait.axera)28 29## Support Platform30- AX650/AX885031  - [M4N-Dock(爱芯派Pro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html)32  - [M.2 Accelerator card](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html)33 34## How to use35 36Download all files from this repository to the device.37 38```39(py310) axera@dell:~/samples/LivePortrait$ tree -L 240.41├── assets42│   └── examples43├── config.json44├── python45│   ├── axmodels46│   ├── cropper.py47│   ├── infer_onnx.py48│   ├── infer.py49│   ├── pretrained_weights50│   ├── requirements.txt51│   └── utils52└── README.md53 547 directories, 6 files55```56 57### python env requirement58 59#### pyaxengine60 61https://github.com/AXERA-TECH/pyaxengine62 63```64wget https://github.com/AXERA-TECH/pyaxengine/releases/download/0.1.3.rc1/axengine-0.1.3-py3-none-any.whl65pip install axengine-0.1.3-py3-none-any.whl66```67 68#### others69 70```71pip install -r python/requirements.txt72```73 74## Inference with AX650 or AX8850 Host, such as AX650 DEMO BOARD, M4N-DOCK(爱芯派Pro)75 76```77root@ax650 ~/yongqiang/LivePortrait.axera # python3 ./python/infer.py --source ./assets/examples/source/s0.jpg --driving ./assets/examples/driving/d8.jpg --models ./python/axmodels/ --output-dir ./axmodel_infer78[INFO] Available providers:  ['AxEngineExecutionProvider']79[INFO] Using provider: AxEngineExecutionProvider80[INFO] Chip type: ChipType.MC5081[INFO] VNPU type: VNPUType.DISABLED82[INFO] Engine version: 2.12.0s83[INFO] Model type: 2 (triple core)84[INFO] Compiler version: 3.3 144960ad85[INFO] Using provider: AxEngineExecutionProvider86[INFO] Model type: 2 (triple core)87[INFO] Compiler version: 3.3 144960ad88[INFO] Using provider: AxEngineExecutionProvider89[INFO] Model type: 2 (triple core)90[INFO] Compiler version: 3.3 0f7260e891[INFO] Using provider: AxEngineExecutionProvider92[INFO] Model type: 2 (triple core)93[INFO] Compiler version: 3.3 144960ad94FaceAnalysisDIY warmup time: 0.598s95LandmarkRunner warmup time: 0.769s962025-05-30 09:56:12.247 | INFO     | __main__:main:727 - Start making driving motion template...972025-05-30 09:56:14.770 | INFO     | __main__:main:747 - Prepared pasteback mask done.982025-05-30 09:56:17.219 | INFO     | __main__:main:787 - The output of image-driven portrait animation is an image.992025-05-30 09:56:30.701 | DEBUG    | __main__:warp_decode:647 - warp time: 13.475s1002025-05-30 09:56:31.118 | INFO     | __main__:main:881 - Animated image: ./axmodel_infer/s0--d8.jpg1012025-05-30 09:56:31.118 | INFO     | __main__:main:882 - Animated image with concat: ./axmodel_infer/s0--d8_concat.jpg1022025-05-30 09:56:31.167 | DEBUG    | __main__:<module>:894 - LivePortrait axmodel infer time: 32.455s103```104 105## Inference with M.2 Accelerator card106 107[What is M.2 Accelerator card?](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html), Show this DEMO based on x86.108 109### Image110 111```112(py310) axera@dell:~/samples/LivePortrait$ python ./python/infer.py --source ./assets/examples/source/s0.jpg --driving ./assets/examples/driving/d8.jpg --models ./python/axmodels/ --output-dir ./axmodel_infer113[INFO] Available providers:  ['AXCLRTExecutionProvider']114[INFO] Using provider: AXCLRTExecutionProvider115[INFO] SOC Name: AX650N116[INFO] VNPU type: VNPUType.DISABLED117[INFO] Compiler version: 3.3 144960ad118[INFO] Using provider: AXCLRTExecutionProvider119[INFO] SOC Name: AX650N120[INFO] VNPU type: VNPUType.DISABLED121[INFO] Compiler version: 3.3 144960ad122[INFO] Using provider: AXCLRTExecutionProvider123[INFO] SOC Name: AX650N124[INFO] VNPU type: VNPUType.DISABLED125[INFO] Compiler version: 3.3 0f7260e8126[INFO] Using provider: AXCLRTExecutionProvider127[INFO] SOC Name: AX650N128[INFO] VNPU type: VNPUType.DISABLED129[INFO] Compiler version: 3.3 144960ad130FaceAnalysisDIY warmup time: 0.024s131[20:02:20] LandmarkRunner warmup time: 0.031s    human_landmark_runner.py:951322025-05-29 20:02:20.727 | INFO     | __main__:main:727 - Start making driving motion template...1332025-05-29 20:02:20.972 | INFO     | __main__:main:747 - Prepared pasteback mask done.1342025-05-29 20:02:21.449 | INFO     | __main__:main:787 - The output of image-driven portrait animation is an image.1352025-05-29 20:02:25.475 | DEBUG    | __main__:warp_decode:647 - warp time: 4.017s1362025-05-29 20:02:25.892 | INFO     | __main__:main:881 - Animated image: ./axmodel_infer/s0--d8.jpg1372025-05-29 20:02:25.892 | INFO     | __main__:main:882 - Animated image with concat: ./axmodel_infer/s0--d8_concat.jpg1382025-05-29 20:02:25.904 | DEBUG    | __main__:<module>:894 - LivePortrait axmodel infer time: 8.165s139(py310) axera@dell:~/samples/LivePortrait$140```141 142Here, `--models` specifies the storage path for the `*.axmodel model`.143 144The output of axmodel-infer is as follows:145 146![output_concat](assets/examples/result/s0--d8_concat_axmodel.jpg)147![output](assets/examples/result/s0--d8_axmodel.jpg)148 149### Video150 151```152python3 ./python/infer.py --source ./assets/examples/source/s0.jpg --driving ./assets/examples/driving/d0.mp4 --models ./python/axmodels/ --output-dir ./axmodel_infer153```154 155The output of `axmodel-infer` is as follows:156 157![output_concat](assets/examples/result/01.gif)158![output](assets/examples/result/02.gif)159