hololens/stable-diffusion-webui-depthmap-script

1# Documentation
2
3## Python scripts
4
5These files are for our monocular 3D Tracking pipeline:
6
7`main.py` Execute 3D photo inpainting
8
9`mesh.py` Functions about context-aware depth inpainting
10
11`mesh_tools.py` Some common functions used in `mesh.py`
12
13`utils.py` Some common functions used in image preprocessing, data loading
14
15`networks.py` Network architectures of inpainting model
16
17
18MiDaS/
19
20`run.py` Execute depth estimation
21
22`monodepth_net.py` Network architecture of depth estimation model
23
24`MiDaS_utils.py` Some common functions in depth estimation
25
26
27## Configuration
28
29```bash
30argument.yml
31```
32
33- `depth_edge_model_ckpt: checkpoints/EdgeModel.pth`
34    - Pretrained model of depth-edge inpainting
35- `depth_feat_model_ckpt: checkpoints/DepthModel.pth`
36    - Pretrained model of depth inpainting
37- `rgb_feat_model_ckpt: checkpoints/ColorModel.pth`
38    - Pretrained model of color inpainting
39- `MiDaS_model_ckpt: MiDaS/model.pt`
40    - Pretrained model of depth estimation
41- `use_boostmonodepth: True`
42    - Use [BoostMonocularDepth](https://github.com/compphoto/BoostingMonocularDepth) to get sharper monocular depth estimation
43- `fps: 40`
44    - Frame per second of output rendered video
45- `num_frames: 240`
46    - Total number of frames in output rendered video
47- `x_shift_range: [-0.03, -0.03, -0.03]`
48    - The translations on x-axis of output rendered videos.
49    - This parameter is a list. Each element corresponds to a specific camera motion.
50- `y_shift_range: [-0.00, -0.00, -0.03]`
51    - The translations on y-axis of output rendered videos.
52    - This parameter is a list. Each element corresponds to a specific camera motion.
53- `z_shift_range: [-0.07, -0.07, -0.07]`
54    - The translations on z-axis of output rendered videos.
55    - This parameter is a list. Each element corresponds to a specific camera motion.
56- `traj_types: ['straight-line', 'circle', 'circle']`
57    - The type of camera trajectory.
58    - This parameter is a list.
59    - Currently, we only privode `straight-line` and `circle`.
60-  `video_postfix: ['zoom-in', 'swing', 'circle']`
61    - The postfix of video.
62    - This parameter is a list.
63- Note that the number of elements in `x_shift_range`,  `y_shift_range`, `z_shift_range`, `traj_types` and `video_postfix` should be equal.
64- `specific: '' `
65    - The specific image name, use this to specify the image to be executed. By default, all the image in the folder will    be executed.
66- `longer_side_len: 960`
67    - The length of larger dimension in output resolution.
68- `src_folder: image`
69    - Input image directory. 
70- `depth_folder: depth`
71    - Estimated depth directory.
72- `mesh_folder: mesh`
73    - Output 3-D mesh directory.
74- `video_folder: video`
75    - Output rendered video directory
76- `load_ply: False`
77    - Action to load existed mesh (.ply) file
78- `save_ply: True`
79    - Action to store the output mesh (.ply) file
80    - Disable this option `save_ply: False` to reduce the computational time.
81- `inference_video: True`
82    - Action to rendered the output video
83- `gpu_ids: 0`
84    - The ID of working GPU. Leave it blank or negative to use CPU.
85- `offscreen_rendering: True`
86    - If you're executing the process in a remote server (via ssh), please switch on this flag. 
87    - Sometimes, using off-screen rendering result in longer execution time.
88- `img_format: '.jpg'`
89    - Input image format.
90- `depth_format: '.npy'`
91    - Input depth (disparity) format. Use NumPy array file as default.
92    - If the user wants to edit the depth (disparity) map manually, we provide `.png` format depth (disparity) map.
93        - Remember to switch this parameter from `.npy` to `.png` when using depth (disparity) map with `.png` format.
94- `require_midas: True`
95    - Set it to `True` if the user wants to use depth map estimated by `MiDaS`.
96    - Set it to `False` if the user wants to use manually edited depth map.
97    - If the user wants to edit the depth (disparity) map manually, we provide `.png` format depth (disparity) map.
98        - Remember to switch this parameter from `True` to `False` when using manually edited depth map.
99- `depth_threshold: 0.04`
100    - A threshold in disparity, adjacent two pixels are discontinuity pixels 
101      if the difference between them excceed this number.
102- `ext_edge_threshold: 0.002`
103    - The threshold to define inpainted depth edge. A pixel in inpainted edge 
104      map belongs to extended depth edge if the value of that pixel exceeds this number,
105- `sparse_iter: 5`
106    - Total iteration numbers of bilateral median filter
107- `filter_size: [7, 7, 5, 5, 5]`
108    - Window size of bilateral median filter in each iteration.
109- `sigma_s: 4.0`
110    - Intensity term of bilateral median filter
111- `sigma_r: 0.5`
112    - Spatial term of bilateral median filter
113- `redundant_number: 12`
114    - The number defines short segments. If a depth edge is shorter than this number, 
115      it is a short segment and removed.
116- `background_thickness: 70`
117    - The thickness of synthesis area.
118- `context_thickness: 140`
119    - The thickness of context area.
120- `background_thickness_2: 70`
121    - The thickness of synthesis area when inpaint second time.
122- `context_thickness_2: 70`
123    - The thickness of context area when inpaint second time.
124- `discount_factor: 1.00`
125- `log_depth: True`
126    - The scale of depth inpainting. If true, performing inpainting in log scale. 
127      Otherwise, performing in linear scale.
128- `largest_size: 512`
129    - The largest size of inpainted image patch.
130- `depth_edge_dilate: 10`
131    - The thickness of dilated synthesis area.
132- `depth_edge_dilate_2: 5`
133    - The thickness of dilated synthesis area when inpaint second time.
134- `extrapolate_border: True`
135    - Action to extrapolate out-side the border.
136- `extrapolation_thickness: 60`
137    - The thickness of extrapolated area.
138- `repeat_inpaint_edge: True`
139    - Action to apply depth edge inpainting model repeatedly. Sometimes inpainting depth 
140      edge once results in short inpinated edge, apply depth edge inpainting repeatedly 
141      could help you prolong the inpainted depth edge. 
142- `crop_border: [0.03, 0.03, 0.05, 0.03]`
143    - The fraction of pixels to crop out around the borders `[top, left, bottom, right]`.
144- `anti_flickering: True`
145    - Action to avoid flickering effect in the output video. 
146    - This may result in longer computational time in rendering phase.
147