timm/MobileCLIP2-S4-OpenCLIP

1---2tags:3- clip4- mobileclip25library_name: open_clip6pipeline_tag: zero-shot-image-classification7license: apple-amlr8---9# Model card for MobileCLIP2-S4-OpenCLIP10These weights and model card are adapted from the original Apple model at https://huggingface.co/apple/MobileCLIP2-S4. This version uses canonical OpenCLIP configs and weight naming.11 12MobileCLIP2 was introduced in [MobileCLIP2: Improving Multi-Modal Reinforced Training](http://arxiv.org/abs/2508.20691) (TMLR August 2025 <mark>Featured</mark>), by Fartash Faghri, Pavan Kumar Anasosalu Vasu, Cem Koc, Vaishaal Shankar, Alexander T Toshev, Oncel Tuzel, Hadi Pouransari.13 14This repository contains the **MobileCLIP2-S4** checkpoint.15 16### Highlights17 18* `MobileCLIP2-S4` matches the accuracy of SigLIP-SO400M/14 with 2x fewer parameters and surpasses DFN ViT-L/14 at 2.5x lower latency measured on iPhone12 Pro Max.19* `MobileCLIP-S3/S4` are our new architectures trained on MobileCLIP’s training dataset, DataCompDR-1B (dashed lines).20* Our smallest variant `MobileCLIP-S0` obtains similar zero-shot performance as [OpenAI](https://arxiv.org/abs/2103.00020)'s ViT-B/16 model while being 4.8x faster and 2.8x smaller.21* `MobileCLIP-S2` obtains better avg zero-shot performance than [SigLIP](https://arxiv.org/abs/2303.15343)'s ViT-B/16 model while being 2.3x faster and 2.1x smaller, and trained with 3x less seen samples.22* `MobileCLIP-B (LT)` attains zero-shot ImageNet performance of **77.2%** which is significantly better than recent works like [DFN](https://arxiv.org/abs/2309.17425) and [SigLIP](https://arxiv.org/abs/2303.15343) with similar architectures or even [OpenAI's ViT-L/14@336](https://arxiv.org/abs/2103.00020).23 24 25## Checkpoints and Results (Original Apple links)26 27| Model                                                     | # Seen <BR>Samples (B) | # Params (M) <BR> (img + txt) | Latency (ms) <BR> (img + txt) | IN-1k Zero-Shot <BR> Top-1 Acc. (%) | Avg. Perf. (%) <BR> on 38 datasets |28|:----------------------------------------------------------|:----------------------:|:-----------------------------:|:-----------------------------:|:-----------------------------------:|:----------------------------------:|29| [MobileCLIP2-S0](https://hf.co/apple/MobileCLIP2-S0)      |           13           |          11.4 + 42.4          |           1.5 + 1.6           |               71.5                 |                59.7                 |30| [MobileCLIP2-S2](https://hf.co/apple/MobileCLIP2-S2)      |           13           |          35.7 + 63.4          |           3.6 + 3.3           |               77.2                 |                64.1                 |31| [MobileCLIP2-B](https://hf.co/apple/MobileCLIP2-B)        |           13           |          86.3 + 63.4          |          10.4 + 3.3           |               79.4                 |                65.8                 |32| [MobileCLIP2-S3](https://hf.co/apple/MobileCLIP2-S3)      |           13           |         125.1 + 123.6         |           8.0 + 6.6           |               80.7                 |                66.8                 |33| [MobileCLIP2-L/14](https://hf.co/apple/MobileCLIP2-L-14)  |           13           |         304.3 + 123.6         |          57.9 + 6.6           |               81.9                 |                67.8                 |34| [MobileCLIP2-S4](https://hf.co/apple/MobileCLIP2-S4)      |           13           |         321.6 + 123.6         |          19.6 + 6.6           |               81.9                 |                67.5                 |35| [MobileCLIP-S0](https://hf.co/apple/MobileCLIP-S0)        |           13           |          11.4 + 42.4          |           1.5 + 1.6           |               67.8                 |                58.1                 |36| [MobileCLIP-S1](https://hf.co/apple/MobileCLIP-S1)        |           13           |          21.5 + 63.4          |           2.5 + 3.3           |               72.6                 |                61.3                 |37| [MobileCLIP-S2](https://hf.co/apple/MobileCLIP-S2)        |           13           |          35.7 + 63.4          |           3.6 + 3.3           |               74.4                 |                63.7                 |38| [MobileCLIP-B](https://hf.co/apple/MobileCLIP-B)          |           13           |          86.3 + 63.4          |          10.4 + 3.3           |               76.8                 |                65.2                 |39| [MobileCLIP-B (LT)](https://hf.co/apple/MobileCLIP-B-LT)  |           36           |          86.3 + 63.4          |          10.4 + 3.3           |               77.2                 |                65.8                 |40| [MobileCLIP-S3](https://hf.co/apple/MobileCLIP-S3)        |           13           |         125.1 + 123.6         |           8.0 + 6.6           |               78.3                 |                66.3                 |41| [MobileCLIP-L/14](https://hf.co/apple/MobileCLIP-L-14)    |           13           |         304.3 + 123.6         |          57.9 + 6.6           |               79.5                 |                66.9                 |42| [MobileCLIP-S4](https://hf.co/apple/MobileCLIP-S4)        |           13           |         321.6 + 123.6         |          19.6 + 6.6           |               79.4                 |                68.1                 |43 44 45## How to Use46 47```py48import torch49import open_clip50from PIL import Image51from urllib.request import urlopen52from timm.utils import reparameterize_model53 54model, _, preprocess = open_clip.create_model_and_transforms('MobileCLIP2-S4', pretrained='dfndr2b')55model.eval()56tokenizer = open_clip.get_tokenizer('MobileCLIP2-S4')57 58# For inference/model exporting purposes, optionally reparameterize for better performance59model = reparameterize_model(model)60 61image = Image.open(urlopen(62    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'63))64image = preprocess(image).unsqueeze(0)65text = tokenizer(["a diagram", "a dog", "a cat", "a doughnut"])66 67with torch.no_grad(), torch.amp.autocast(image.device.type):68    image_features = model.encode_image(image)69    text_features = model.encode_text(text)70    image_features /= image_features.norm(dim=-1, keepdim=True)71    text_features /= text_features.norm(dim=-1, keepdim=True)72    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)73 74print("Label probs:", text_probs)75```