nvidia/C-RADIO

1---2license: other3license_name: nvidia-open-model-license4license_link: >-5  https://huggingface.co/nvidia/C-RADIO/resolve/main/nvidia-open-model-license-agreement-june-2024.pdf6library_name: transformers7---8 9# Model Overview10 11## Description:12This model performs visual feature extraction.13For instance, RADIO generates image embeddings that can be used by a downstream model to classify images.14 15### License/Terms of Use16 17[License] This model is governed by the [NVIDIA Open Model License Agreement](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf).18 19## References:20 21[**AM-RADIO: Agglomerative Vision Foundation Model - Reduce All Domains Into One**](https://arxiv.org/abs/2312.06709)22 23[**PHI-S: Distribution Balancing for Label-Free Multi-Teacher Distillation**](https://arxiv.org/abs/2410.01680)24 25[**RADIO Amplified: Improved Baselines for Agglomerative Vision Foundation Models**](https://arxiv.org/pdf/2412.07679)26 27 28## Model Architecture:29**Architecture Type:** Neural Network  <br>30**Network Architecture:** Vision Transformer <br>31 32## Input:33**Input Type(s):** Image <br>34**Input Format(s):** Red, Green, Blue (RGB) pixel values in [0, 1] range. <br>35**Input Parameters:** Two Dimensional (2D) <br>36**Other Properties Related to Input:** Image resolutions up to 2048x2028 in increments of 16 pixels <br>37 38## Output:39**Output Type(s):** Embeddings <br>40**Output Format:** Tensor <br>41**Output Parameters:** 2D <br>42**Other Properties Related to Output:** Downstream model required to leverage image features <br>43 44## Usage:45 46RADIO will return a tuple with two tensors.47The `summary` is similar to the `cls_token` in ViT and is meant to represent the general concept of the entire image.48It has shape `(B,C)` with `B` being the batch dimension, and `C` being some number of channels.49The `spatial_features` represent more localized content which should be suitable for dense tasks such as semantic segmentation, or for integration into an LLM.50 51```python52import torch53from PIL import Image54from transformers import AutoModel, CLIPImageProcessor55 56hf_repo = "nvidia/C-RADIO"57 58image_processor = CLIPImageProcessor.from_pretrained(hf_repo)59model = AutoModel.from_pretrained(hf_repo, trust_remote_code=True)60model.eval().cuda()61 62image = Image.open('./assets/radio.png').convert('RGB')63pixel_values = image_processor(images=image, return_tensors='pt', do_resize=True).pixel_values64pixel_values = pixel_values.cuda()65 66summary, features = model(pixel_values)67```68 69Spatial features have shape `(B,T,D)` with `T` being the flattened spatial tokens, and `D` being the channels for spatial features. Note that `C!=D` in general.70Converting to a spatial tensor format can be done using the downsampling size of the model, combined with the input tensor shape. For RADIO, the patch size is 16.71 72```Python73from einops import rearrange74spatial_features = rearrange(spatial_features, 'b (h w) d -> b d h w', h=x.shape[-2] // patch_size, w=x.shape[-1] // patch_size)75```76 77The resulting tensor will have shape `(B,D,H,W)`, as is typically seen with computer vision models.78 79 80## Software Integration:81**Runtime Engine(s):**82* TAO- 24.10 <br>83 84**Supported Hardware Microarchitecture Compatibility:** <br>85* NVIDIA Ampere <br>86* NVIDIA Blackwell <br>87* NVIDIA Jetson  <br>88* NVIDIA Hopper <br>89* NVIDIA Lovelace <br>90* NVIDIA Pascal <br>91* NVIDIA Turing <br>92* NVIDIA Volta <br>93 94**[Preferred/Supported] Operating System(s):** <br>95* Linux96* Linux 4 Tegra97* QNX98* Windows99 100## Model Version(s):101C-RADIO.102 103**Link:** https://huggingface.co/nvidia/C-RADIO104 105# Training, Testing, and Evaluation Datasets:106 107## Training Dataset:108 109NV-CC-Img-Text-Dataset  <br>110** Data Collection Method by dataset <br>111* Automated <br>112** Labeling Method by dataset <br>113* Not Applicable (no labels are needed) <br>114**Properties:** 700 Million Images <br>115 116## Evaluation Dataset:117**Link:** [ImageNet](https://www.image-net.org/)  <br>118** Data Collection Method by dataset <br>119* Automated <br>120** Labeling Method by dataset <br>121* Human <br>122 123**Properties:** This dataset spans 1000 object classes and contains 1,281,167 training images, 50,000 validation images and 100,000 test images.<br>124 125## Inference:126**Engine:** PyTorch <br>127**Test Hardware:** A100 <br>128 129## Ethical Considerations (For NVIDIA Models Only):130NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications.  When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. Users should evaluate the model for safety and quality for a specific use case and build additional guardrails as appropriate.131 132Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).