hololens/stable-diffusion-webui-depthmap-script
1
1<div align="center">
2<h1>Depth Anything V2</h1>
3
4[**Lihe Yang**](https://liheyoung.github.io/)<sup>1</sup> · [**Bingyi Kang**](https://bingykang.github.io/)<sup>2†</sup> · [**Zilong Huang**](http://speedinghzl.github.io/)<sup>2</sup>
5<br>
6[**Zhen Zhao**](http://zhaozhen.me/) · [**Xiaogang Xu**](https://xiaogang00.github.io/) · [**Jiashi Feng**](https://sites.google.com/site/jshfeng/)<sup>2</sup> · [**Hengshuang Zhao**](https://hszhao.github.io/)<sup>1*</sup>
7
8<sup>1</sup>HKU   <sup>2</sup>TikTok
9<br>
10†project lead *corresponding author
11†[Bingyi Kang](https://bingykang.github.io/) proposed this project and advised in every aspect.
12
13<a href="https://arxiv.org/abs/2406.09414"><img src='https://img.shields.io/badge/arXiv-Depth Anything V2-red' alt='Paper PDF'></a>
14<a href='https://depth-anything-v2.github.io'><img src='https://img.shields.io/badge/Project_Page-Depth Anything V2-green' alt='Project Page'></a>
15<a href='https://huggingface.co/spaces/depth-anything/Depth-Anything-V2'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Demo-blue'></a>
16<a href='https://huggingface.co/datasets/depth-anything/DA-2K'><img src='https://img.shields.io/badge/Benchmark-DA--2K-yellow' alt='Benchmark'></a>
17</div>
18
19This work presents Depth Anything V2. It significantly outperforms [V1](https://github.com/LiheYoung/Depth-Anything) in fine-grained details and robustness. Compared with SD-based models, it enjoys faster inference speed, fewer parameters, and higher depth accuracy.
20
21
22
23
24## News
25
26- **2024-07-06:** Depth Anything V2 is supported in [Transformers](https://github.com/huggingface/transformers/). See the [instructions](https://huggingface.co/docs/transformers/main/en/model_doc/depth_anything_v2) for convenient usage.
27- **2024-06-25:** Depth Anything is integrated into [Apple Core ML Models](https://developer.apple.com/machine-learning/models/). See the instructions ([V1](https://huggingface.co/apple/coreml-depth-anything-small), [V2](https://huggingface.co/apple/coreml-depth-anything-v2-small)) for usage.
28- **2024-06-22:** We release [smaller metric depth models](https://github.com/DepthAnything/Depth-Anything-V2/tree/main/metric_depth#pre-trained-models) based on Depth-Anything-V2-Small and Base.
29- **2024-06-20:** Our repository and project page are flagged by GitHub and removed from the public for 6 days. Sorry for the inconvenience.
30- **2024-06-14:** Paper, project page, code, models, demo, and benchmark are all released.
31
32
33## Pre-trained Models
34
35We provide **four models** of varying scales for robust relative depth estimation:
36
37| Model | Params | Checkpoint |
38|:-|-:|:-:|
39| Depth-Anything-V2-Small | 24.8M | [Download](https://huggingface.co/depth-anything/Depth-Anything-V2-Small/resolve/main/depth_anything_v2_vits.pth?download=true) |
40| Depth-Anything-V2-Base | 97.5M | [Download](https://huggingface.co/depth-anything/Depth-Anything-V2-Base/resolve/main/depth_anything_v2_vitb.pth?download=true) |
41| Depth-Anything-V2-Large | 335.3M | [Download](https://huggingface.co/depth-anything/Depth-Anything-V2-Large/resolve/main/depth_anything_v2_vitl.pth?download=true) |
42| Depth-Anything-V2-Giant | 1.3B | Coming soon |
43
44
45## Usage
46
47### Prepraration
48
49```bash
50git clone https://github.com/DepthAnything/Depth-Anything-V2
51cd Depth-Anything-V2
52pip install -r requirements.txt
53```
54
55Download the checkpoints listed [here](#pre-trained-models) and put them under the `checkpoints` directory.
56
57### Use our models
58```python
59import cv2
60import torch
61
62from depth_anything_v2.dpt import DepthAnythingV2
63
64DEVICE = 'cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu'
65
66model_configs = {
67 'vits': {'encoder': 'vits', 'features': 64, 'out_channels': [48, 96, 192, 384]},
68 'vitb': {'encoder': 'vitb', 'features': 128, 'out_channels': [96, 192, 384, 768]},
69 'vitl': {'encoder': 'vitl', 'features': 256, 'out_channels': [256, 512, 1024, 1024]},
70 'vitg': {'encoder': 'vitg', 'features': 384, 'out_channels': [1536, 1536, 1536, 1536]}
71}
72
73encoder = 'vitl' # or 'vits', 'vitb', 'vitg'
74
75model = DepthAnythingV2(**model_configs[encoder])
76model.load_state_dict(torch.load(f'checkpoints/depth_anything_v2_{encoder}.pth', map_location='cpu'))
77model = model.to(DEVICE).eval()
78
79raw_img = cv2.imread('your/image/path')
80depth = model.infer_image(raw_img) # HxW raw depth map in numpy
81```
82
83If you do not want to clone this repository, you can also load our models through [Transformers](https://github.com/huggingface/transformers/). Below is a simple code snippet. Please refer to the [official page](https://huggingface.co/docs/transformers/main/en/model_doc/depth_anything_v2) for more details.
84
85- Note 1: Make sure you can connect to Hugging Face and have installed the latest Transformers.
86- Note 2: Due to the [upsampling difference](https://github.com/huggingface/transformers/pull/31522#issuecomment-2184123463) between OpenCV (we used) and Pillow (HF used), predictions may differ slightly. So you are more recommended to use our models through the way introduced above.
87```python
88from transformers import pipeline
89from PIL import Image
90
91pipe = pipeline(task="depth-estimation", model="depth-anything/Depth-Anything-V2-Small-hf")
92image = Image.open('your/image/path')
93depth = pipe(image)["depth"]
94```
95
96### Running script on *images*
97
98```bash
99python run.py \
100 --encoder <vits | vitb | vitl | vitg> \
101 --img-path <path> --outdir <outdir> \
102 [--input-size <size>] [--pred-only] [--grayscale]
103```
104Options:
105- `--img-path`: You can either 1) point it to an image directory storing all interested images, 2) point it to a single image, or 3) point it to a text file storing all image paths.
106- `--input-size` (optional): By default, we use input size `518` for model inference. ***You can increase the size for even more fine-grained results.***
107- `--pred-only` (optional): Only save the predicted depth map, without raw image.
108- `--grayscale` (optional): Save the grayscale depth map, without applying color palette.
109
110For example:
111```bash
112python run.py --encoder vitl --img-path assets/examples --outdir depth_vis
113```
114
115### Running script on *videos*
116
117```bash
118python run_video.py \
119 --encoder <vits | vitb | vitl | vitg> \
120 --video-path assets/examples_video --outdir video_depth_vis \
121 [--input-size <size>] [--pred-only] [--grayscale]
122```
123
124***Our larger model has better temporal consistency on videos.***
125
126### Gradio demo
127
128To use our gradio demo locally:
129
130```bash
131python app.py
132```
133
134You can also try our [online demo](https://huggingface.co/spaces/Depth-Anything/Depth-Anything-V2).
135
136***Note: Compared to V1, we have made a minor modification to the DINOv2-DPT architecture (originating from this [issue](https://github.com/LiheYoung/Depth-Anything/issues/81)).*** In V1, we *unintentionally* used features from the last four layers of DINOv2 for decoding. In V2, we use [intermediate features](https://github.com/DepthAnything/Depth-Anything-V2/blob/2cbc36a8ce2cec41d38ee51153f112e87c8e42d8/depth_anything_v2/dpt.py#L164-L169) instead. Although this modification did not improve details or accuracy, we decided to follow this common practice.
137
138
139## Fine-tuned to Metric Depth Estimation
140
141Please refer to [metric depth estimation](./metric_depth).
142
143
144## DA-2K Evaluation Benchmark
145
146Please refer to [DA-2K benchmark](./DA-2K.md).
147
148
149## Community Support
150
151**We sincerely appreciate all the community support for our Depth Anything series. Thank you a lot!**
152
153- Apple Core ML:
154 - https://developer.apple.com/machine-learning/models
155 - https://huggingface.co/apple/coreml-depth-anything-v2-small
156 - https://huggingface.co/apple/coreml-depth-anything-small
157- Transformers:
158 - https://huggingface.co/docs/transformers/main/en/model_doc/depth_anything_v2
159 - https://huggingface.co/docs/transformers/main/en/model_doc/depth_anything
160- TensorRT:
161 - https://github.com/spacewalk01/depth-anything-tensorrt
162 - https://github.com/zhujiajian98/Depth-Anythingv2-TensorRT-python
163- ONNX: https://github.com/fabio-sim/Depth-Anything-ONNX
164- ComfyUI: https://github.com/kijai/ComfyUI-DepthAnythingV2
165- Transformers.js (real-time depth in web): https://huggingface.co/spaces/Xenova/webgpu-realtime-depth-estimation
166- Android:
167 - https://github.com/shubham0204/Depth-Anything-Android
168 - https://github.com/FeiGeChuanShu/ncnn-android-depth_anything
169
170
171## Acknowledgement
172
173We are sincerely grateful to the awesome Hugging Face team ([@Pedro Cuenca](https://huggingface.co/pcuenq), [@Niels Rogge](https://huggingface.co/nielsr), [@Merve Noyan](https://huggingface.co/merve), [@Amy Roberts](https://huggingface.co/amyeroberts), et al.) for their huge efforts in supporting our models in Transformers and Apple Core ML.
174
175We also thank the [DINOv2](https://github.com/facebookresearch/dinov2) team for contributing such impressive models to our community.
176
177
178## LICENSE
179
180Depth-Anything-V2-Small model is under the Apache-2.0 license. Depth-Anything-V2-Base/Large/Giant models are under the CC-BY-NC-4.0 license.
181
182
183## Citation
184
185If you find this project useful, please consider citing:
186
187```bibtex
188@article{depth_anything_v2,
189 title={Depth Anything V2},
190 author={Yang, Lihe and Kang, Bingyi and Huang, Zilong and Zhao, Zhen and Xu, Xiaogang and Feng, Jiashi and Zhao, Hengshuang},
191 journal={arXiv:2406.09414},
192 year={2024}
193}
194
195@inproceedings{depth_anything_v1,
196 title={Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data},
197 author={Yang, Lihe and Kang, Bingyi and Huang, Zilong and Xu, Xiaogang and Feng, Jiashi and Zhao, Hengshuang},
198 booktitle={CVPR},
199 year={2024}
200}
201```
202