google/tipsv1-s14

1---2license: apache-2.03tags:4- vision5- image-text6- contrastive-learning7- zero-shot8- feature-extraction9- arxiv:2410.1651210library_name: transformers11pipeline_tag: zero-shot-image-classification12---13 14# TIPS — S/14 (v1)15 16TIPS (Text-Image Pre-training with Spatial awareness, ICLR 2025) is a family of contrastive vision-language models that produce spatially rich image features aligned with text embeddings. This is the original (v1) S/14 release with 22M vision params and 34M text params, converted from the [official checkpoints](https://github.com/google-deepmind/tips).17 18| Variant | Vision params | Text params | Embed dim | Resolution |19|---------|---------------|-------------|-----------|------------|20| [S/14](https://huggingface.co/google/tipsv1-s14) | 22M | 34M | 384 | 448 |21| [B/14](https://huggingface.co/google/tipsv1-b14) | 86M | 110M | 768 | 448 |22| [L/14](https://huggingface.co/google/tipsv1-l14) | 304M | 184M | 1024 | 448 |23| [So400m/14](https://huggingface.co/google/tipsv1-so400m14) | 413M | 448M | 1152 | 448 |24| [g/14](https://huggingface.co/google/tipsv1-g14) | 1.1B | 389M | 1536 | 448 |25| [g/14 low-res](https://huggingface.co/google/tipsv1-g14-lowres) | 1.1B | 389M | 1536 | 224 |26 27## Usage28 29```bash30pip install transformers torch torchvision sentencepiece scikit-learn requests31```32 33### Load the model34 35```python36from transformers import AutoModel37 38model = AutoModel.from_pretrained("google/tipsv1-s14", trust_remote_code=True)39model.eval()40```41 42### Encode images43 44Images should be tensors in `[0, 1]` range (just `ToTensor()`, no ImageNet normalization).45 46```python47import requests48from PIL import Image49from torchvision import transforms50 51url = "https://huggingface.co/spaces/google/TIPSv2/resolve/main/examples/zeroseg/pascal_context_00049_image.png"52image = Image.open(requests.get(url, stream=True).raw).convert("RGB")53transform = transforms.Compose([transforms.Resize((448, 448)), transforms.ToTensor()])54pixel_values = transform(image).unsqueeze(0)55 56out = model.encode_image(pixel_values)57print(out.cls_token.shape)     # (1, 1, 384) — global image embedding58print(out.patch_tokens.shape)  # (1, 1024, 384) — per-patch spatial features59```60 61The second CLS token (`out.register_tokens`) was trained on synthetic captions; the first (`out.cls_token`) on web alt-text, and is the one aligned with the text tower.62 63### Encode text64 65```python66text_emb = model.encode_text(["a photo of a bus", "a photo of a dog"])67print(text_emb.shape)  # (2, 384) — one embedding per query68```69 70### Zero-shot classification71 72```python73import torch.nn.functional as F74 75classes = ["bus", "car", "dog", "cat"]76cls = F.normalize(out.cls_token[:, 0, :], dim=-1)77text_emb = F.normalize(model.encode_text(classes), dim=-1)78similarity = cls @ text_emb.T79print(classes[similarity.argmax()])  # predicted class80```81 82### Visualize spatial features83 84```python85import numpy as np86from sklearn.decomposition import PCA87 88feat = out.patch_tokens[0].detach().cpu().numpy()89rgb = PCA(n_components=3, whiten=True).fit_transform(feat).reshape(32, 32, 3)90rgb = 1 / (1 + np.exp(-2.0 * rgb))  # sigmoid for [0, 1] range with good contrast91```92 93## Model details94 95- ViT-S/14 vision encoder (12 layers, patch size 14, two CLS tokens) + 12-layer transformer text encoder96- Native resolution 448; other patch-multiple resolutions work via positional-embedding interpolation97- Preprocessing: images to `[0, 1]`, no normalization; SentencePiece tokenizer, lowercased, max 64 tokens98 99## License100 101Apache 2.0102 103## Citation104 105```bibtex106@inproceedings{maninis2025tips,107  title     = {{TIPS: Text-Image Pretraining with Spatial Awareness}},108  author    = {Maninis, Kevis-Kokitsi and Chen, Kaifeng and Ghosh, Soham and Karpur, Arjun and Chen, Koert and Xia, Ye and Cao, Bingyi and Salz, Daniel and Han, Guangxing and Dlabal, Jan and Gnanapragasam, Dan and Seyedhosseini, Mojtaba and Zhou, Howard and Araujo, Andre},109  booktitle = {International Conference on Learning Representations (ICLR)},110  year      = {2025},111  url       = {https://arxiv.org/abs/2410.16512}112}113```114