stack86/CodeFormer

1<p align="center">2  <img src="assets/CodeFormer_logo.png" height=110>3</p>4 5## Towards Robust Blind Face Restoration with Codebook Lookup Transformer6 7[Paper](https://arxiv.org/abs/2206.11253) | [Project Page](https://shangchenzhou.com/projects/CodeFormer/) | [Video](https://youtu.be/d3VDpkXlueI)8 9 10<a href="https://colab.research.google.com/drive/1m52PNveE4PBhYrecj34cnpEeiHcC5LTb?usp=sharing"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="google colab logo"></a> [![Replicate](https://img.shields.io/badge/Demo-%F0%9F%9A%80%20Replicate-blue)](https://replicate.com/sczhou/codeformer) ![visitors](https://visitor-badge.glitch.me/badge?page_id=sczhou/CodeFormer)11 12[Shangchen Zhou](https://shangchenzhou.com/), [Kelvin C.K. Chan](https://ckkelvinchan.github.io/), [Chongyi Li](https://li-chongyi.github.io/), [Chen Change Loy](https://www.mmlab-ntu.com/person/ccloy/) 13 14S-Lab, Nanyang Technological University15 16<img src="assets/network.jpg" width="800px"/>17 18 19:star: If CodeFormer is helpful to your images or projects, please help star this repo. Thanks! :hugs: 20 21### Update22 23- **2022.09.09**: Integrated to :rocket: [Replicate](https://replicate.com/). Try out online demo! [![Replicate](https://img.shields.io/badge/Demo-%F0%9F%9A%80%20Replicate-blue)](https://replicate.com/sczhou/codeformer)24- **2022.09.04**: Add face upsampling `--face_upsample` for high-resolution AI-created face enhancement.25- **2022.08.23**: Some modifications on face detection and fusion for better AI-created face enhancement.26- **2022.08.07**: Integrate [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN) to support background image enhancement.27- **2022.07.29**: Integrate new face detectors of `['RetinaFace'(default), 'YOLOv5']`. 28- **2022.07.17**: Add Colab demo of CodeFormer. <a href="https://colab.research.google.com/drive/1m52PNveE4PBhYrecj34cnpEeiHcC5LTb?usp=sharing"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="google colab logo"></a>29- **2022.07.16**: Release inference code for face restoration. :blush:30- **2022.06.21**: This repo is created.31 32### TODO33- [ ] Add checkpoint for face inpainting34- [ ] Add training code and config files35- [x] ~~Add background image enhancement~~36 37#### Face Restoration38 39<img src="assets/restoration_result1.png" width="400px"/> <img src="assets/restoration_result2.png" width="400px"/>40<img src="assets/restoration_result3.png" width="400px"/> <img src="assets/restoration_result4.png" width="400px"/>41 42#### Face Color Enhancement and Restoration43 44<img src="assets/color_enhancement_result1.png" width="400px"/> <img src="assets/color_enhancement_result2.png" width="400px"/>45 46#### Face Inpainting47 48<img src="assets/inpainting_result1.png" width="400px"/> <img src="assets/inpainting_result2.png" width="400px"/>49 50 51 52### Dependencies and Installation53 54- Pytorch >= 1.7.155- CUDA >= 10.156- Other required packages in `requirements.txt`57```58# git clone this repository59git clone https://github.com/sczhou/CodeFormer60cd CodeFormer61 62# create new anaconda env63conda create -n codeformer python=3.8 -y64conda activate codeformer65 66# install python dependencies67pip3 install -r requirements.txt68python basicsr/setup.py develop69```70<!-- conda install -c conda-forge dlib -->71 72### Quick Inference73 74##### Download Pre-trained Models:75Download the facelib pretrained models from [[Google Drive](https://drive.google.com/drive/folders/1b_3qwrzY_kTQh0-SnBoGBgOrJ_PLZSKm?usp=sharing) | [OneDrive](https://entuedu-my.sharepoint.com/:f:/g/personal/s200094_e_ntu_edu_sg/EvDxR7FcAbZMp_MA9ouq7aQB8XTppMb3-T0uGZ_2anI2mg?e=DXsJFo)] to the `weights/facelib` folder. You can manually download the pretrained models OR download by runing the following command.76```77python scripts/download_pretrained_models.py facelib78```79 80Download the CodeFormer pretrained models from [[Google Drive](https://drive.google.com/drive/folders/1CNNByjHDFt0b95q54yMVp6Ifo5iuU6QS?usp=sharing) | [OneDrive](https://entuedu-my.sharepoint.com/:f:/g/personal/s200094_e_ntu_edu_sg/EoKFj4wo8cdIn2-TY2IV6CYBhZ0pIG4kUOeHdPR_A5nlbg?e=AO8UN9)] to the `weights/CodeFormer` folder. You can manually download the pretrained models OR download by runing the following command.81```82python scripts/download_pretrained_models.py CodeFormer83```84 85##### Prepare Testing Data:86You can put the testing images in the `inputs/TestWhole` folder. If you would like to test on cropped and aligned faces, you can put them in the `inputs/cropped_faces` folder.87 88 89##### Testing on Face Restoration:90```91# For cropped and aligned faces92python inference_codeformer.py --w 0.5 --has_aligned --test_path [input folder]93 94# For the whole images95# Add '--bg_upsampler realesrgan' to enhance the background regions with Real-ESRGAN96# Add '--face_upsample' to further upsample restorated face with Real-ESRGAN97python inference_codeformer.py --w 0.7 --test_path [input folder]98```99 100NOTE that *w* is in [0, 1]. Generally, smaller *w* tends to produce a higher-quality result, while larger *w* yields a higher-fidelity result. 101 102The results will be saved in the `results` folder.103 104### Citation105If our work is useful for your research, please consider citing:106 107    @article{zhou2022codeformer,108        author = {Zhou, Shangchen and Chan, Kelvin C.K. and Li, Chongyi and Loy, Chen Change},109        title = {Towards Robust Blind Face Restoration with Codebook Lookup TransFormer},110        journal = {arXiv preprint arXiv:2206.11253},111        year = {2022}112    }113 114### License115 116<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License</a>.117 118### Acknowledgement119 120This project is based on [BasicSR](https://github.com/XPixelGroup/BasicSR). We also borrow some codes from [Unleashing Transformers](https://github.com/samb-t/unleashing-transformers), [YOLOv5-face](https://github.com/deepcam-cn/yolov5-face), and [FaceXLib](https://github.com/xinntao/facexlib). Thanks for their awesome works.121 122### Contact123If you have any question, please feel free to reach me out at `shangchenzhou@gmail.com`.