Intel/dpt-large

1---2license: apache-2.03tags:4- vision5- depth-estimation6widget:7- src: https://huggingface.co/datasets/mishig/sample_images/resolve/main/tiger.jpg8  example_title: Tiger9- src: https://huggingface.co/datasets/mishig/sample_images/resolve/main/teapot.jpg10  example_title: Teapot11- src: https://huggingface.co/datasets/mishig/sample_images/resolve/main/palace.jpg12  example_title: Palace13 14model-index:15- name: dpt-large16  results:17  - task:18      type: monocular-depth-estimation19      name: Monocular Depth Estimation20    dataset:21      type: MIX-622      name: MIX-623    metrics:24    - type: Zero-shot transfer25      value: 10.8226      name: Zero-shot transfer27      config: Zero-shot transfer28      verified: false29---30 31## Model Details: DPT-Large (also known as MiDaS 3.0)32 33Dense Prediction Transformer (DPT) model trained on 1.4 million images for monocular depth estimation. 34It was introduced in the paper [Vision Transformers for Dense Prediction](https://arxiv.org/abs/2103.13413) by Ranftl et al. (2021) and first released in [this repository](https://github.com/isl-org/DPT). 35DPT uses the Vision Transformer (ViT) as backbone and adds a neck + head on top for monocular depth estimation.36![model image](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/dpt_architecture.jpg)37 38The model card has been written in combination by the Hugging Face team and Intel.39 40| Model Detail | Description |41| ----------- | ----------- | 42| Model Authors - Company | Intel | 43| Date | March 22, 2022 | 44| Version | 1 | 45| Type | Computer Vision - Monocular Depth Estimation | 46| Paper or Other Resources | [Vision Transformers for Dense Prediction](https://arxiv.org/abs/2103.13413) and [GitHub Repo](https://github.com/isl-org/DPT) | 47| License | Apache 2.0 |48| Questions or Comments | [Community Tab](https://huggingface.co/Intel/dpt-large/discussions) and [Intel Developers Discord](https://discord.gg/rv2Gp55UJQ)|49 50| Intended Use | Description |51| ----------- | ----------- | 52| Primary intended uses | You can use the raw model for zero-shot monocular depth estimation. See the [model hub](https://huggingface.co/models?search=dpt) to look for fine-tuned versions on a task that interests you. | 53| Primary intended users | Anyone doing monocular depth estimation | 54| Out-of-scope uses | This model in most cases will need to be fine-tuned for your particular task.  The model should not be used to intentionally create hostile or alienating environments for people.|55 56 57### How to use58 59The easiest is leveraging the pipeline API:60 61```62from transformers import pipeline63 64pipe = pipeline(task="depth-estimation", model="Intel/dpt-large")65result = pipe(image)66result["depth"]67```68 69In case you want to implement the entire logic yourself, here's how to do that for zero-shot depth estimation on an image:70 71```python72from transformers import DPTImageProcessor, DPTForDepthEstimation73import torch74import numpy as np75from PIL import Image76import requests77 78url = "http://images.cocodataset.org/val2017/000000039769.jpg"79image = Image.open(requests.get(url, stream=True).raw)80 81processor = DPTImageProcessor.from_pretrained("Intel/dpt-large")82model = DPTForDepthEstimation.from_pretrained("Intel/dpt-large")83 84# prepare image for the model85inputs = processor(images=image, return_tensors="pt")86 87with torch.no_grad():88    outputs = model(**inputs)89    predicted_depth = outputs.predicted_depth90 91# interpolate to original size92prediction = torch.nn.functional.interpolate(93    predicted_depth.unsqueeze(1),94    size=image.size[::-1],95    mode="bicubic",96    align_corners=False,97)98 99# visualize the prediction100output = prediction.squeeze().cpu().numpy()101formatted = (output * 255 / np.max(output)).astype("uint8")102depth = Image.fromarray(formatted)103```104 105For more code examples, we refer to the [documentation](https://huggingface.co/docs/transformers/master/en/model_doc/dpt).106 107 108| Factors | Description | 109| ----------- | ----------- | 110| Groups | Multiple datasets compiled together | 111| Instrumentation | - |112| Environment | Inference completed on Intel Xeon Platinum 8280 CPU @ 2.70GHz with 8 physical cores and an NVIDIA RTX 2080 GPU. |113| Card Prompts | Model deployment on alternate hardware and software will change model performance |114 115| Metrics | Description | 116| ----------- | ----------- | 117| Model performance measures | Zero-shot Transfer |118| Decision thresholds | - | 119| Approaches to uncertainty and variability | - | 120 121| Training and Evaluation Data | Description | 122| ----------- | ----------- | 123| Datasets | The dataset is called MIX 6, and contains around 1.4M images. The model was initialized with ImageNet-pretrained weights.|124| Motivation | To build a robust monocular depth prediction network |125| Preprocessing | "We resize the image such that the longer side is 384 pixels and train on random square crops of size 384. ... We perform random horizontal flips for data augmentation." See [Ranftl et al. (2021)](https://arxiv.org/abs/2103.13413) for more details. | 126 127## Quantitative Analyses128| Model | Training set | DIW WHDR | ETH3D AbsRel | Sintel AbsRel | KITTI δ>1.25 | NYU δ>1.25 | TUM δ>1.25 |129| --- | --- | --- | --- | --- | --- | --- | --- | 130| DPT - Large | MIX 6 | 10.82 (-13.2%) | 0.089 (-31.2%) | 0.270 (-17.5%) | 8.46 (-64.6%) | 8.32 (-12.9%) | 9.97 (-30.3%) |131| DPT - Hybrid | MIX 6 | 11.06 (-11.2%) | 0.093 (-27.6%) | 0.274 (-16.2%) | 11.56 (-51.6%) | 8.69 (-9.0%) | 10.89 (-23.2%) | 132| MiDaS  | MIX 6  | 12.95 (+3.9%)  | 0.116 (-10.5%)  | 0.329 (+0.5%)  | 16.08 (-32.7%)  | 8.71 (-8.8%)  | 12.51 (-12.5%)133| MiDaS [30]  | MIX 5  | 12.46  | 0.129  | 0.327  | 23.90  | 9.55  | 14.29 | 134 | Li [22]  | MD [22]  | 23.15  | 0.181  | 0.385  | 36.29  | 27.52  | 29.54 | 135 | Li [21]  | MC [21]  | 26.52  | 0.183  | 0.405  | 47.94  | 18.57  | 17.71 | 136 | Wang [40]  | WS [40]  | 19.09  | 0.205  | 0.390  | 31.92  | 29.57  | 20.18 | 137 | Xian [45]  | RW [45]  | 14.59  | 0.186 |  0.422  | 34.08 |  27.00 |  25.02 | 138 | Casser [5]  | CS [8]  | 32.80  | 0.235  | 0.422  | 21.15  | 39.58  | 37.18 | 139 140Table 1. Comparison to the state of the art on monocular depth estimation. We evaluate zero-shot cross-dataset transfer according to the141protocol defined in [30]. Relative performance is computed with respect to the original MiDaS model [30]. Lower is better for all metrics. ([Ranftl et al., 2021](https://arxiv.org/abs/2103.13413))142 143 144| Ethical Considerations | Description | 145| ----------- | ----------- | 146| Data | The training data come from multiple image datasets compiled together. |147| Human life | The model is not intended to inform decisions central to human life or flourishing. It is an aggregated set of monocular depth image datasets. | 148| Mitigations | No additional risk mitigation strategies were considered during model development. |149| Risks and harms | The extent of the risks involved by using the model remain unknown. |150| Use cases | - | 151 152| Caveats and Recommendations |153| ----------- | 154| Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. There are no additional caveats or recommendations for this model. |155 156 157 158### BibTeX entry and citation info159 160```bibtex161@article{DBLP:journals/corr/abs-2103-13413,162  author    = {Ren{\'{e}} Ranftl and163               Alexey Bochkovskiy and164               Vladlen Koltun},165  title     = {Vision Transformers for Dense Prediction},166  journal   = {CoRR},167  volume    = {abs/2103.13413},168  year      = {2021},169  url       = {https://arxiv.org/abs/2103.13413},170  eprinttype = {arXiv},171  eprint    = {2103.13413},172  timestamp = {Wed, 07 Apr 2021 15:31:46 +0200},173  biburl    = {https://dblp.org/rec/journals/corr/abs-2103-13413.bib},174  bibsource = {dblp computer science bibliography, https://dblp.org}175}176```