qualcomm/OpenAI-Clip
109.9k
1---2library_name: pytorch3license: mit4tags:5- foundation6- android7pipeline_tag: image-classification8 9---10 1112 13# OpenAI-Clip: Optimized for Qualcomm Devices14 15Contrastive Language-Image Pre-Training (CLIP) uses a ViT like transformer to get visual features and a causal language model to get the text features. Both the text and visual features can then be used for a variety of zero-shot learning tasks.16 17This is based on the implementation of OpenAI-Clip found [here](https://github.com/openai/CLIP/).18This repository contains pre-exported model files optimized for Qualcomm® devices. You can use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/v0.63.0/src/qai_hub_models/models/openai_clip) library to export with custom configurations. More details on model performance across various devices, can be found [here](#performance-summary).19 20Qualcomm AI Hub Models uses [Qualcomm AI Hub Workbench](https://workbench.aihub.qualcomm.com) to compile, profile, and evaluate this model. [Sign up](https://myaccount.qualcomm.com/signup) to run these models on a hosted Qualcomm® device.21 22## Getting Started23There are two ways to deploy this model on your device:24 25### Option 1: Download Pre-Exported Models26 27Below are pre-exported model assets ready for deployment.28 29| Runtime | Precision | Chipset | SDK Versions | Download |30|---|---|---|---|---|31| ONNX | float | Universal | QAIRT 2.50, ONNX Runtime 1.27.1 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/openai_clip/releases/v0.63.0/openai_clip-onnx-float.zip)32| ONNX | w8a16 | Universal | QAIRT 2.50, ONNX Runtime 1.27.1 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/openai_clip/releases/v0.63.0/openai_clip-onnx-w8a16.zip)33| QNN_DLC | float | Universal | QAIRT 2.50 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/openai_clip/releases/v0.63.0/openai_clip-qnn_dlc-float.zip)34| QNN_DLC | w8a16 | Universal | QAIRT 2.50 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/openai_clip/releases/v0.63.0/openai_clip-qnn_dlc-w8a16.zip)35| TFLITE | float | Universal | QAIRT 2.50 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/openai_clip/releases/v0.63.0/openai_clip-tflite-float.zip)36 37For more device-specific assets and performance metrics, visit **[OpenAI-Clip on Qualcomm® AI Hub](https://aihub.qualcomm.com/models/openai_clip)**.38 39 40### Option 2: Export with Custom Configurations41 42Use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/v0.63.0/src/qai_hub_models/models/openai_clip) Python library to compile and export the model with your own:43- Custom weights (e.g., fine-tuned checkpoints)44- Custom input shapes45- Target device and runtime configurations46 47This option is ideal if you need to customize the model beyond the default configuration provided here.48 49See our repository for [OpenAI-Clip on GitHub](https://github.com/qualcomm/ai-hub-models/blob/v0.63.0/src/qai_hub_models/models/openai_clip) for usage instructions.50 51## Model Details52 53**Model Type:** Model_use_case.image_classification54 55**Model Stats:**56- Image input resolution: 224x22457- Model checkpoint: ViT-B/1658- Model size (float): 571 MB59- Number of parameters: 150M60- Text context length: 7761 62## Performance Summary63| Model | Runtime | Precision | Chipset | Inference Time (ms) | Peak Memory Range (MB) | Primary Compute Unit64|---|---|---|---|---|---|---65| OpenAI-Clip | ONNX | float | Snapdragon® 8 Elite Gen 5 For Galaxy Mobile | 13.245 ms | 1 - 613 MB | NPU66| OpenAI-Clip | ONNX | float | Snapdragon® 8 Elite For Galaxy Mobile | 16.662 ms | 0 - 602 MB | NPU67| OpenAI-Clip | ONNX | float | Snapdragon® X2 Elite | 13.948 ms | 2 - 2 MB | NPU68| OpenAI-Clip | ONNX | float | Snapdragon® X Elite | 32.635 ms | 294 - 294 MB | NPU69| OpenAI-Clip | ONNX | float | Snapdragon® 8 Gen 3 Mobile | 21.83 ms | 1 - 992 MB | NPU70| OpenAI-Clip | ONNX | float | Snapdragon® 8 Gen 1 Mobile | 41.33 ms | 1 - 848 MB | NPU71| OpenAI-Clip | ONNX | float | Qualcomm® Dragonwing™ IQ-8275 | 35.502 ms | 0 - 5 MB | NPU72| OpenAI-Clip | ONNX | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 31.987 ms | 0 - 322 MB | NPU73| OpenAI-Clip | ONNX | float | Qualcomm® QCS8450 | 41.33 ms | 1 - 848 MB | NPU74| OpenAI-Clip | ONNX | float | Qualcomm® Dragonwing™ IQ-9075 | 36.392 ms | 1 - 5 MB | NPU75| OpenAI-Clip | ONNX | float | Qualcomm® Dragonwing™ IQ-X7181 | 32.635 ms | 294 - 294 MB | NPU76| OpenAI-Clip | ONNX | float | Qualcomm® Dragonwing™ Q-8750 | 16.662 ms | 0 - 602 MB | NPU77| OpenAI-Clip | ONNX | w8a16 | Snapdragon® 8 Elite Gen 5 For Galaxy Mobile | 11.485 ms | 0 - 658 MB | NPU78| OpenAI-Clip | ONNX | w8a16 | Snapdragon® 8 Elite For Galaxy Mobile | 14.918 ms | 0 - 620 MB | NPU79| OpenAI-Clip | ONNX | w8a16 | Snapdragon® X2 Elite | 11.929 ms | 1 - 1 MB | NPU80| OpenAI-Clip | ONNX | w8a16 | Snapdragon® X Elite | 25.222 ms | 173 - 173 MB | NPU81| OpenAI-Clip | ONNX | w8a16 | Snapdragon® 8 Gen 3 Mobile | 17.896 ms | 0 - 739 MB | NPU82| OpenAI-Clip | ONNX | w8a16 | Snapdragon® 8 Gen 1 Mobile | 34.933 ms | 0 - 740 MB | NPU83| OpenAI-Clip | ONNX | w8a16 | Qualcomm® Dragonwing™ QCS6490 | 103.718 ms | 3 - 6 MB | NPU84| OpenAI-Clip | ONNX | w8a16 | Qualcomm® Dragonwing™ IQ-8275 | 23.545 ms | 0 - 5 MB | NPU85| OpenAI-Clip | ONNX | w8a16 | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 24.639 ms | 0 - 195 MB | NPU86| OpenAI-Clip | ONNX | w8a16 | Qualcomm® QCS8450 | 34.933 ms | 0 - 740 MB | NPU87| OpenAI-Clip | ONNX | w8a16 | Qualcomm® Dragonwing™ IQ-9075 | 24.63 ms | 0 - 4 MB | NPU88| OpenAI-Clip | ONNX | w8a16 | Qualcomm® Dragonwing™ IQ-X7181 | 25.222 ms | 173 - 173 MB | NPU89| OpenAI-Clip | ONNX | w8a16 | Qualcomm® Dragonwing™ Q-6690 | 334.61 ms | 0 - 690 MB | NPU90| OpenAI-Clip | ONNX | w8a16 | Qualcomm® Dragonwing™ Q-7790 | 37.089 ms | 1 - 638 MB | NPU91| OpenAI-Clip | ONNX | w8a16 | Qualcomm® Dragonwing™ Q-8750 | 14.918 ms | 0 - 620 MB | NPU92| OpenAI-Clip | ONNX | w8a16 | Snapdragon® 7 Gen 4 Mobile | 37.089 ms | 1 - 638 MB | NPU93| OpenAI-Clip | QNN_DLC | float | Snapdragon® 8 Elite Gen 5 For Galaxy Mobile | 13.098 ms | 0 - 489 MB | NPU94| OpenAI-Clip | QNN_DLC | float | Snapdragon® 8 Elite For Galaxy Mobile | 16.747 ms | 1 - 483 MB | NPU95| OpenAI-Clip | QNN_DLC | float | Snapdragon® X2 Elite | 14.143 ms | 1 - 1 MB | NPU96| OpenAI-Clip | QNN_DLC | float | Snapdragon® X Elite | 33.128 ms | 1 - 1 MB | NPU97| OpenAI-Clip | QNN_DLC | float | Snapdragon® 8 Gen 3 Mobile | 21.394 ms | 0 - 688 MB | NPU98| OpenAI-Clip | QNN_DLC | float | Snapdragon® 8 Gen 1 Mobile | 37.296 ms | 1 - 815 MB | NPU99| OpenAI-Clip | QNN_DLC | float | Qualcomm® Dragonwing™ IQ-8275 | 35.973 ms | 1 - 4 MB | NPU100| OpenAI-Clip | QNN_DLC | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 31.39 ms | 1 - 3 MB | NPU101| OpenAI-Clip | QNN_DLC | float | Qualcomm® SA8650P | 36.675 ms | 1 - 700 MB | NPU102| OpenAI-Clip | QNN_DLC | float | Qualcomm® SA8255P | 36.675 ms | 1 - 700 MB | NPU103| OpenAI-Clip | QNN_DLC | float | Qualcomm® QCS8450 | 37.296 ms | 1 - 815 MB | NPU104| OpenAI-Clip | QNN_DLC | float | Qualcomm® Dragonwing™ IQ-9075 | 36.419 ms | 3 - 5 MB | NPU105| OpenAI-Clip | QNN_DLC | float | Qualcomm® Dragonwing™ IQ-X7181 | 33.128 ms | 1 - 1 MB | NPU106| OpenAI-Clip | QNN_DLC | float | Qualcomm® Dragonwing™ Q-8750 | 16.747 ms | 1 - 483 MB | NPU107| OpenAI-Clip | QNN_DLC | float | Qualcomm® SA8295P | 37.723 ms | 1 - 591 MB | NPU108| OpenAI-Clip | QNN_DLC | w8a16 | Snapdragon® 8 Elite Gen 5 For Galaxy Mobile | 11.506 ms | 0 - 628 MB | NPU109| OpenAI-Clip | QNN_DLC | w8a16 | Snapdragon® 8 Elite For Galaxy Mobile | 15.226 ms | 0 - 591 MB | NPU110| OpenAI-Clip | QNN_DLC | w8a16 | Snapdragon® X2 Elite | 12.1 ms | 0 - 0 MB | NPU111| OpenAI-Clip | QNN_DLC | w8a16 | Snapdragon® X Elite | 28.154 ms | 0 - 0 MB | NPU112| OpenAI-Clip | QNN_DLC | w8a16 | Snapdragon® 8 Gen 3 Mobile | 18.724 ms | 0 - 702 MB | NPU113| OpenAI-Clip | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ IQ-8275 | 24.827 ms | 0 - 3 MB | NPU114| OpenAI-Clip | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 27.141 ms | 0 - 440 MB | NPU115| OpenAI-Clip | QNN_DLC | w8a16 | Qualcomm® SA8775P | 26.821 ms | 0 - 566 MB | NPU116| OpenAI-Clip | QNN_DLC | w8a16 | Qualcomm® SA8650P | 26.821 ms | 0 - 566 MB | NPU117| OpenAI-Clip | QNN_DLC | w8a16 | Qualcomm® SA8255P | 26.821 ms | 0 - 566 MB | NPU118| OpenAI-Clip | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ IQ-9075 | 27.154 ms | 0 - 3 MB | NPU119| OpenAI-Clip | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ IQ-X7181 | 28.154 ms | 0 - 0 MB | NPU120| OpenAI-Clip | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ Q-6690 | 432.665 ms | 1 - 693 MB | NPU121| OpenAI-Clip | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ Q-7790 | 37.332 ms | 0 - 611 MB | NPU122| OpenAI-Clip | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ Q-8750 | 15.226 ms | 0 - 591 MB | NPU123| OpenAI-Clip | QNN_DLC | w8a16 | Qualcomm® SA7255P | 58.663 ms | 0 - 564 MB | NPU124| OpenAI-Clip | QNN_DLC | w8a16 | Snapdragon® 7 Gen 4 Mobile | 37.332 ms | 0 - 611 MB | NPU125| OpenAI-Clip | TFLITE | float | Snapdragon® 8 Elite Gen 5 For Galaxy Mobile | 12.822 ms | 0 - 499 MB | NPU126| OpenAI-Clip | TFLITE | float | Snapdragon® 8 Elite For Galaxy Mobile | 17.003 ms | 0 - 490 MB | NPU127| OpenAI-Clip | TFLITE | float | Snapdragon® 8 Gen 3 Mobile | 21.13 ms | 0 - 693 MB | NPU128| OpenAI-Clip | TFLITE | float | Snapdragon® 8 Gen 1 Mobile | 36.247 ms | 0 - 677 MB | NPU129| OpenAI-Clip | TFLITE | float | Qualcomm® Dragonwing™ IQ-8275 | 34.563 ms | 0 - 297 MB | NPU130| OpenAI-Clip | TFLITE | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 30.974 ms | 0 - 3 MB | NPU131| OpenAI-Clip | TFLITE | float | Qualcomm® SA8650P | 35.976 ms | 0 - 670 MB | NPU132| OpenAI-Clip | TFLITE | float | Qualcomm® SA8255P | 35.976 ms | 0 - 670 MB | NPU133| OpenAI-Clip | TFLITE | float | Qualcomm® QCS8450 | 36.247 ms | 0 - 677 MB | NPU134| OpenAI-Clip | TFLITE | float | Qualcomm® Dragonwing™ IQ-9075 | 35.657 ms | 0 - 297 MB | NPU135| OpenAI-Clip | TFLITE | float | Qualcomm® Dragonwing™ Q-8750 | 17.003 ms | 0 - 490 MB | NPU136| OpenAI-Clip | TFLITE | float | Qualcomm® SA7255P | 748.41 ms | 11 - 35 MB | CPU137| OpenAI-Clip | TFLITE | float | Qualcomm® SA8295P | 37.294 ms | 0 - 441 MB | NPU138 139## License140* The license for the original implementation of OpenAI-Clip can be found141 [here](https://github.com/openai/CLIP/blob/main/LICENSE).142 143## References144* [Learning Transferable Visual Models From Natural Language Supervision](https://arxiv.org/abs/2103.00020)145* [Source Model Implementation](https://github.com/openai/CLIP/)146 147## Community148* Join [our AI Hub Slack community](https://aihub.qualcomm.com/community/slack) to collaborate, post questions and learn more about on-device AI.149* For questions or feedback please [reach out to us](mailto:ai-hub-support@qti.qualcomm.com).150