hololens/stable-diffusion-webui-depthmap-script
1
1# [CVPR 2020] 3D Photography using Context-aware Layered Depth Inpainting
2
3[](https://colab.research.google.com/drive/1706ToQrkIZshRSJSHvZ1RuCiM__YX3Bz)
4
5### [[Paper](https://arxiv.org/abs/2004.04727)] [[Project Website](https://shihmengli.github.io/3D-Photo-Inpainting/)] [[Google Colab](https://colab.research.google.com/drive/1706ToQrkIZshRSJSHvZ1RuCiM__YX3Bz)]
6
7<p align='center'>
8<img src='https://filebox.ece.vt.edu/~jbhuang/project/3DPhoto/3DPhoto_teaser.jpg' width='900'/>
9</p>
10
11We propose a method for converting a single RGB-D input image into a 3D photo, i.e., a multi-layer representation for novel view synthesis that contains hallucinated color and depth structures in regions occluded in the original view. We use a Layered Depth Image with explicit pixel connectivity as underlying representation, and present a learning-based inpainting model that iteratively synthesizes new local color-and-depth content into the occluded region in a spatial context-aware manner. The resulting 3D photos can be efficiently rendered with motion parallax using standard graphics engines. We validate the effectiveness of our method on a wide range of challenging everyday scenes and show fewer artifacts when compared with the state-of-the-arts.
12<br/>
13
14**3D Photography using Context-aware Layered Depth Inpainting**
15<br/>
16[Meng-Li Shih](https://shihmengli.github.io/),
17[Shih-Yang Su](https://lemonatsu.github.io/),
18[Johannes Kopf](https://johanneskopf.de/), and
19[Jia-Bin Huang](https://filebox.ece.vt.edu/~jbhuang/)
20<br/>
21In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
22
23
24## Prerequisites
25
26- Linux (tested on Ubuntu 18.04.4 LTS)
27- Anaconda
28- Python 3.7 (tested on 3.7.4)
29- PyTorch 1.4.0 (tested on 1.4.0 for execution)
30
31and the Python dependencies listed in [requirements.txt](requirements.txt)
32- To get started, please run the following commands:
33 ```bash
34 conda create -n 3DP python=3.7 anaconda
35 conda activate 3DP
36 pip install -r requirements.txt
37 conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit==10.1.243 -c pytorch
38 ```
39- Next, please download the model weight using the following command:
40 ```bash
41 chmod +x download.sh
42 ./download.sh
43 ```
44
45## Quick start
46Please follow the instructions in this section.
47This should allow to execute our results.
48For more detailed instructions, please refer to [`DOCUMENTATION.md`](DOCUMENTATION.md).
49
50## Execute
511. Put ```.jpg``` files (e.g., test.jpg) into the ```image``` folder.
52 - E.g., `image/moon.jpg`
532. Run the following command
54 ```bash
55 python main.py --config argument.yml
56 ```
57 - Note: The 3D photo generation process usually takes about 2-3 minutes depending on the available computing resources.
583. The results are stored in the following directories:
59 - Corresponding depth map estimated by [MiDaS](https://github.com/intel-isl/MiDaS.git)
60 - E.g. ```depth/moon.npy```, ```depth/moon.png```
61 - User could edit ```depth/moon.png``` manually.
62 - Remember to set the following two flags as listed below if user wants to use manually edited ```depth/moon.png``` as input for 3D Photo.
63 - `depth_format: '.png'`
64 - `require_midas: False`
65 - Inpainted 3D mesh (Optional: User need to switch on the flag `save_ply`)
66 - E.g. ```mesh/moon.ply```
67 - Rendered videos with zoom-in motion
68 - E.g. ```video/moon_zoom-in.mp4```
69 - Rendered videos with swing motion
70 - E.g. ```video/moon_swing.mp4```
71 - Rendered videos with circle motion
72 - E.g. ```video/moon_circle.mp4```
73 - Rendered videos with dolly zoom-in effect
74 - E.g. ```video/moon_dolly-zoom-in.mp4```
75 - Note: We assume that the object of focus is located at the center of the image.
764. (Optional) If you want to change the default configuration. Please read [`DOCUMENTATION.md`](DOCUMENTATION.md) and modified ```argument.yml```.
77
78
79## License
80This work is licensed under MIT License. See [LICENSE](LICENSE) for details.
81
82If you find our code/models useful, please consider citing our paper:
83```
84@inproceedings{Shih3DP20,
85 author = {Shih, Meng-Li and Su, Shih-Yang and Kopf, Johannes and Huang, Jia-Bin},
86 title = {3D Photography using Context-aware Layered Depth Inpainting},
87 booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
88 year = {2020}
89}
90```
91
92## Acknowledgments
93- We thank Pratul Srinivasan for providing clarification of the method [Srinivasan et al. CVPR 2019](https://people.eecs.berkeley.edu/~pratul/publication/mpi_extrapolation/).
94- We thank the author of [Zhou et al. 2018](https://people.eecs.berkeley.edu/~tinghuiz/projects/mpi/), [Choi et al. 2019](https://github.com/NVlabs/extreme-view-synth/), [Mildenhall et al. 2019](https://github.com/Fyusion/LLFF), [Srinivasan et al. 2019](https://github.com/google-research/google-research/tree/ac9b04e1dbdac468fda53e798a326fe9124e49fe/mpi_extrapolation), [Wiles et al. 2020](http://www.robots.ox.ac.uk/~ow/synsin.html), [Niklaus et al. 2019](https://github.com/sniklaus/3d-ken-burns) for providing their implementations online.
95- Our code builds upon [EdgeConnect](https://github.com/knazeri/edge-connect), [MiDaS](https://github.com/intel-isl/MiDaS.git) and [pytorch-inpainting-with-partial-conv](https://github.com/naoto0804/pytorch-inpainting-with-partial-conv)
96