Veritone/siglip2-base-patch16-224

1---2license: apache-2.03tags:4- vision5widget:6  - src: >-7      https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg8    candidate_labels: bee in the sky, bee on the flower9    example_title: Bee10library_name: transformers11pipeline_tag: zero-shot-image-classification12---13 14# SigLIP 2 Base15 16[SigLIP 2](https://huggingface.co/papers/2502.14786) extends the pretraining objective of17[SigLIP](https://huggingface.co/papers/2303.15343) with prior, independently developed techniques18into a unified recipe, for improved semantic understanding, localization, and dense features.19 20## Intended uses21 22You can use the raw model for tasks like zero-shot image classification and23image-text retrieval, or as a vision encoder for VLMs (and other vision tasks).24 25Here is how to use this model to perform zero-shot image classification:26 27```python28from transformers import pipeline29 30# load pipeline31ckpt = "google/siglip2-base-patch16-224"32image_classifier = pipeline(model=ckpt, task="zero-shot-image-classification")33 34# load image and candidate labels35url = "http://images.cocodataset.org/val2017/000000039769.jpg"36candidate_labels = ["2 cats", "a plane", "a remote"]37 38# run inference39outputs = image_classifier(image, candidate_labels)40print(outputs)41```42 43You can encode an image using the Vision Tower like so:44 45```python46import torch47from transformers import AutoModel, AutoProcessor48from transformers.image_utils import load_image49 50# load the model and processor51ckpt = "google/siglip2-base-patch16-224"52model = AutoModel.from_pretrained(ckpt, device_map="auto").eval()53processor = AutoProcessor.from_pretrained(ckpt)54 55# load the image56image = load_image("https://huggingface.co/datasets/merve/coco/resolve/main/val2017/000000000285.jpg")57inputs = processor(images=[image], return_tensors="pt").to(model.device)58 59# run infernece60with torch.no_grad():61    image_embeddings = model.get_image_features(**inputs)    62 63print(image_embeddings.shape)64```65 66For more code examples, we refer to the [siglip documentation](https://huggingface.co/transformers/main/model_doc/siglip.html#).67 68## Training procedure69 70SigLIP 2 adds some clever training objectives on top of SigLIP:71 721. Decoder loss732. Global-local and masked prediction loss743. Aspect ratio and resolution adaptibility 75 76### Training data77 78SigLIP 2 is pre-trained on the WebLI dataset [(Chen et al., 2023)](https://arxiv.org/abs/2209.06794).79 80### Compute81 82The model was trained on up to 2048 TPU-v5e chips.83 84## Evaluation results85 86Evaluation of SigLIP 2 is shown below (taken from the paper).87 88![Evaluation Table](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/sg2-blog/eval_table.png)89 90### BibTeX entry and citation info91 92```bibtex93@misc{tschannen2025siglip2multilingualvisionlanguage,94      title={SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features}, 95      author={Michael Tschannen and Alexey Gritsenko and Xiao Wang and Muhammad Ferjad Naeem and Ibrahim Alabdulmohsin and Nikhil Parthasarathy and Talfan Evans and Lucas Beyer and Ye Xia and Basil Mustafa and Olivier Hénaff and Jeremiah Harmsen and Andreas Steiner and Xiaohua Zhai},96      year={2025},97      eprint={2502.14786},98      archivePrefix={arXiv},99      primaryClass={cs.CV},100      url={https://arxiv.org/abs/2502.14786}, 101}102```103