nvidia/C-RADIOv2-VLM-H
11715
1---2license: other3license_name: nvidia-open-model-license4license_link: https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf5---6 7# Model Overview8 9[[**Github**](https://github.com/NVlabs/RADIO)] [[**CVPR 2025**](https://arxiv.org/abs/2412.07679)] [[**CVPR 2024**](https://arxiv.org/abs/2312.06709)]10 11## Description12 13This model performs visual feature extraction.14For instance, RADIO generates image embeddings that can be used by a downstream model to classify images.15 16C-RADIOv2 models are available in multiple sizes:17* Base (90M parameters).18* Large (320M parameters).19* Huge (653M parameters).20* Gigantic (1.1B parameters).21 22C-RADIOv2 was trained for 1M steps (400k more steps than v1), using inverse frequency sampling for data balancing, and [PHI Standardization](https://arxiv.org/abs/2410.01680) for teacher distribution balancing.23 24This model is ready for commercial/non-commercial use.25 26### License/Terms of Use27 28GOVERNING TERMS: Use of this model is governed by the [NVIDIA Open Model License Agreement](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf).29 30## Deployment Geography31 32Global.33 34## Use Case35 36The embeddings generated by this model are expected to be used by a downstream application.37For example:38 39* Image-level understanding (image classification, curation, etc.).40* Dense processing (semantic segmentation, depth estimation, etc.).41* Integration into a Vision-Language Model.42 43## Release Date44 45Huggingface: 03/26/2025 via [RADIO Collection of Models](https://huggingface.co/collections/nvidia/radio-669f77f1dd6b153f007dd1c6).46 47## References48 49* \[CVPR 2025\] [**RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Models**](https://arxiv.org/abs/2412.07679)50* \[CVPR 2024\] [**AM-RADIO: Agglomerative Vision Foundation Model - Reduce All Domains Into One**](https://arxiv.org/abs/2312.06709)51 52## Model Architecture53 54**Architecture Type:** Neural Network <br>55**Network Architecture:** Vision Transformer <br>56 57## Input58 59**Input Type(s):** Image <br>60**Input Format(s):** Red, Green, Blue (RGB) <br>61**Input Parameters:** Two Dimensional (2D) <br>62**Other Properties Related to Input:** Image resolutions up to 2048x2028 in increments of 16 pixels <br>63 64## Output65 66**Output Type(s):** Embeddings <br>67**Output Format:** Tensor <br>68**Output Parameters:** 2D <br>69**Other Properties Related to Output:** Downstream model required to leverage image features <br>70 71## Usage:72 73RADIO will return a tuple with two tensors.74The `summary` is similar to the `cls_token` in ViT and is meant to represent the general concept of the entire image.75It has shape `(B,C)` with `B` being the batch dimension, and `C` being some number of channels.76The `spatial_features` represent more localized content which should be suitable for dense tasks such as semantic segmentation, or for integration into an LLM.77 78```python79import torch80from PIL import Image81from transformers import AutoModel, CLIPImageProcessor82 83hf_repo = "nvidia/C-RADIOv2-VLM-H"84 85image_processor = CLIPImageProcessor.from_pretrained(hf_repo)86model = AutoModel.from_pretrained(hf_repo, trust_remote_code=True)87model.eval().cuda()88 89image = Image.open('./assets/radio.png').convert('RGB')90pixel_values = image_processor(images=image, return_tensors='pt', do_resize=True).pixel_values91pixel_values = pixel_values.cuda()92 93summary, features = model(pixel_values)94```95 96Spatial features have shape `(B,T,D)` with `T` being the flattened spatial tokens, and `D` being the channels for spatial features. Note that `C!=D` in general.97Converting to a spatial tensor format can be done using the downsampling size of the model, combined with the input tensor shape. For RADIO, the patch size is 16.98 99```Python100from einops import rearrange101spatial_features = rearrange(spatial_features, 'b (h w) d -> b d h w', h=x.shape[-2] // patch_size, w=x.shape[-1] // patch_size)102```103 104The resulting tensor will have shape `(B,D,H,W)`, as is typically seen with computer vision models.105 106## Software Integration107 108**Runtime Engine(s):**109* TAO- 24.10 <br>110 111**Supported Hardware Microarchitecture Compatibility:** <br>112* NVIDIA Ampere <br>113* NVIDIA Blackwell <br>114* NVIDIA Jetson <br>115* NVIDIA Hopper <br>116* NVIDIA Lovelace <br>117* NVIDIA Pascal <br>118* NVIDIA Turing <br>119* NVIDIA Volta <br>120 121**[Preferred/Supported] Operating System(s):** <br>122* Linux123* Linux 4 Tegra124* QNX125* Windows126 127## Model Version(s)128 129* C-RADIOv2-B (90M parameters).130* C-RADIOv2-L (320M parameters).131* C-RADIOv2-H (653M parameters).132* C-RADIOv2-G (1.8B parameters).133 134**Links:**135 136* https://huggingface.co/nvidia/C-RADIOv2-B137* https://huggingface.co/nvidia/C-RADIOv2-L138* https://huggingface.co/nvidia/C-RADIOv2-H139* https://huggingface.co/nvidia/C-RADIOv2-g140 141# Training and Evaluation Datasets142 143## Training Dataset144 145NV-CC-Img-Text-Dataset <br>146 147### Data Collection Method by dataset148 149* Automated <br>150 151### Labeling Method by dataset152 153* Not Applicable (no labels are needed)154 155### Properties156 157* 700 Million Images <br>158 159## Evaluation Dataset160 161**Link:** [ImageNet](https://www.image-net.org/) <br>162 163### Data Collection Method by dataset164 165* Automated <br>166 167### Labeling Method by dataset168 169* Human <br>170 171**Properties:** This dataset spans 1000 object classes and contains 1,281,167 training images, 50,000 validation images and 100,000 test images.<br>172 173## Inference174 175**Engine:** PyTorch <br>176**Test Hardware:** A100 <br>177 178## Ethical Considerations179 180NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.181 182For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards below.183 184Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).185 186### Bias187 188Field | Response189:---------------------------------------------------------------------------------------------------|:---------------190Participation considerations from adversely impacted groups [protected classes](https://www.senate.ca.gov/content/protected-classes) in model design and testing: | None191Measures taken to mitigate against unwanted bias: | None192 193 194### Explainability195 196Field | Response197:------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------198Intended Application & Domain: | Visual Feature Extraction199Model Type: | Vision Transformer200Intended Users: | Developers of downstream vision applications201Output: | Image embeddings202Describe how the model works: | The model takes an image as input, processes the image through multiple transformer blocks, and outputs summary and patch embeddings.203Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of: | Not Applicable204Technical Limitations: | This model generates image embeddings that can be used by a downstream model to, for example, classify images. The downstream model must be trained to leverage the visual embeddings.205Verified to have met prescribed NVIDIA quality standards: | Yes206Performance Metrics: | Image classification accuracy, semantic segmentation mean-over-intersection.207Potential Known Risks: | This model is only tested on input resolutions ranging from 256 to 2048, in increments of 16 pixels. Additionally, the generated embeddings might fail to disambiguate differences that appear evident to humans (e.g. two images showing different breeds of dogs might in fact produce very similar embeddings). Domain-specific evaluation is required for the target application.208Licensing: | [NVIDIA Open Model License](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf)209 210 211### Privacy212 213Field | Response214:----------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------215Generatable or reverse engineerable personal data? | None216Personal data used to create this model? | None217How often is dataset reviewed? | Before Every Release218Is there provenance for all datasets used in training? | Yes219Does data labeling (annotation, metadata) comply with privacy laws? | Yes220Is data compliant with data subject requests for data correction or removal, if such a request was made? | Yes221 222### Safety223 224Field | Response225:---------------------------------------------------|:----------------------------------226Model Application(s): | Generation of visual embeddings227Describe the life critical impact (if present). | Not Applicable228Use Case Restrictions: | Abide by NVIDIA Open Model License Agreement229Model and dataset restrictions: | The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development. Restrictions enforce dataset access during training, and dataset license constraints adhered to.230