HuggingFaceTB/SmolVLM2-2.2B-Instruct
335169k
1---2library_name: transformers3license: apache-2.04datasets:5- HuggingFaceM4/the_cauldron6- HuggingFaceM4/Docmatix7- lmms-lab/LLaVA-OneVision-Data8- lmms-lab/M4-Instruct-Data9- HuggingFaceFV/finevideo10- MAmmoTH-VL/MAmmoTH-VL-Instruct-12M11- lmms-lab/LLaVA-Video-178K12- orrzohar/Video-STaR13- Mutonix/Vript14- TIGER-Lab/VISTA-400K15- Enxin/MovieChat-1K_train16- ShareGPT4Video/ShareGPT4Video17pipeline_tag: image-text-to-text18tags:19- video-text-to-text20language:21- en22base_model:23- HuggingFaceTB/SmolVLM-Instruct24---25 26 27<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/SmolVLM2_banner.png" width="800" height="auto" alt="Image description">28 29# SmolVLM2 2.2B30 31SmolVLM2-2.2B is a lightweight multimodal model designed to analyze video content. The model processes videos, images, and text inputs to generate text outputs - whether answering questions about media files, comparing visual content, or transcribing text from images. Despite its compact size, requiring only 5.2GB of GPU RAM for video inference, it delivers robust performance on complex multimodal tasks. This efficiency makes it particularly well-suited for on-device applications where computational resources may be limited.32## Model Summary33 34- **Developed by:** Hugging Face 🤗35- **Model type:** Multi-modal model (image/multi-image/video/text)36- **Language(s) (NLP):** English37- **License:** Apache 2.038- **Architecture:** Based on [Idefics3](https://huggingface.co/HuggingFaceM4/Idefics3-8B-Llama3) (see technical summary)39 40## Resources41 42- **Demo:** [Video Highlight Generator](https://huggingface.co/spaces/HuggingFaceTB/SmolVLM2-HighlightGenerator)43- **Blog:** [Blog post](https://huggingface.co/blog/smolvlm2)44 45 46## Uses47 48 49SmolVLM2 can be used for inference on multimodal (video / image / text) tasks where the input consists of text queries along with video or one or more images. Text and media files can be interleaved arbitrarily, enabling tasks like captioning, visual question answering, and storytelling based on visual content. The model does not support image or video generation.50 51To fine-tune SmolVLM2 on a specific task, you can follow [the fine-tuning tutorial](https://github.com/huggingface/smollm/blob/main/vision/finetuning/Smol_VLM_FT.ipynb).52 53## Evaluation 54 55### Vision Evaluation56 57| Model | Mathvista | MMMU | OCRBench | MMStar | AI2D | ChartQA_Test | Science_QA | TextVQA Val | DocVQA Val |58|-------------------|-----------|-------|----------|--------|------|--------------|------------|-------------|------------|59| **SmolVLM2 2.2B** | 51.5 | 42 | 72.9 | 46 | 70 | 68.84 | 90 | 73.21 | 79.98 |60| SmolVLM 2.2B | 43.9 | 38.3 | 65.5 | 41.8 | 84.5 | 71.6 | 84.5 | 72.1 | 79.7 |61 62 63### Video Evaluation64We evaluated the performance of the SmolVLM2 family on the following scientific benchmarks:65 66| Size | Video-MME | MLVU | MVBench |67|----------|-----------------|----------|---------------|68| 2.2B | 52.1 | 55.2 | 46.27 |69| 500M | 42.2 | 47.3 | 39.73 |70| 256M | 33.7 | 40.6 | 32.7 |71 72 73### How to get started74 75You can use transformers to load, infer and fine-tune SmolVLM. Make sure you have num2words, flash-attn and latest transformers installed.76You can load the model as follows.77 78```python79from transformers import AutoProcessor, AutoModelForImageTextToText80import torch81 82model_path = "HuggingFaceTB/SmolVLM2-2.2B-Instruct"83processor = AutoProcessor.from_pretrained(model_path)84model = AutoModelForImageTextToText.from_pretrained(85 model_path,86 torch_dtype=torch.bfloat16,87 _attn_implementation="flash_attention_2"88).to("cuda")89```90 91#### Simple Inference92 93You preprocess your inputs directly using chat templates and directly passing them 94 95```python96messages = [97 {98 "role": "user",99 "content": [100 {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"},101 {"type": "text", "text": "Can you describe this image?"},102 ]103 },104]105 106inputs = processor.apply_chat_template(107 messages,108 add_generation_prompt=True,109 tokenize=True,110 return_dict=True,111 return_tensors="pt",112).to(model.device, dtype=torch.bfloat16)113 114generated_ids = model.generate(**inputs, do_sample=False, max_new_tokens=64)115generated_texts = processor.batch_decode(116 generated_ids,117 skip_special_tokens=True,118)119print(generated_texts[0])120```121 122#### Video Inference123 124To use SmolVLM2 for video inference, make sure you have decord installed. 125 126```python127messages = [128 {129 "role": "user",130 "content": [131 {"type": "video", "path": "path_to_video.mp4"},132 {"type": "text", "text": "Describe this video in detail"}133 ]134 },135]136 137inputs = processor.apply_chat_template(138 messages,139 add_generation_prompt=True,140 tokenize=True,141 return_dict=True,142 return_tensors="pt",143).to(model.device, dtype=torch.bfloat16)144 145generated_ids = model.generate(**inputs, do_sample=False, max_new_tokens=64)146generated_texts = processor.batch_decode(147 generated_ids,148 skip_special_tokens=True,149)150 151print(generated_texts[0])152```153#### Multi-image Interleaved Inference154 155You can interleave multiple media with text using chat templates.156 157```python158import torch159 160 161messages = [162 {163 "role": "user",164 "content": [165 {"type": "text", "text": "What is the similarity between these two images?"},166 {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"},167 {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"}, 168 ]169 },170]171 172inputs = processor.apply_chat_template(173 messages,174 add_generation_prompt=True,175 tokenize=True,176 return_dict=True,177 return_tensors="pt",178).to(model.device, dtype=torch.bfloat16)179 180generated_ids = model.generate(**inputs, do_sample=False, max_new_tokens=64)181generated_texts = processor.batch_decode(182 generated_ids,183 skip_special_tokens=True,184)185print(generated_texts[0])186```187 188 189### Model optimizations190 191## Misuse and Out-of-scope Use192 193SmolVLM is not intended for high-stakes scenarios or critical decision-making processes that affect an individual's well-being or livelihood. The model may produce content that appears factual but may not be accurate. Misuse includes, but is not limited to:194 195- Prohibited Uses:196 - Evaluating or scoring individuals (e.g., in employment, education, credit)197 - Critical automated decision-making198 - Generating unreliable factual content199- Malicious Activities:200 - Spam generation201 - Disinformation campaigns202 - Harassment or abuse203 - Unauthorized surveillance204 205### License206 207SmolVLM2 is built upon [the shape-optimized SigLIP](https://huggingface.co/google/siglip-so400m-patch14-384) as image encoder and [SmolLM2](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct) for text decoder part.208 209We release the SmolVLM2 checkpoints under the Apache 2.0 license.210 211## Citation information212You can cite us in the following way:213```bibtex214@article{marafioti2025smolvlm,215 title={SmolVLM: Redefining small and efficient multimodal models}, 216 author={Andrés Marafioti and Orr Zohar and Miquel Farré and Merve Noyan and Elie Bakouch and Pedro Cuenca and Cyril Zakka and Loubna Ben Allal and Anton Lozhkov and Nouamane Tazi and Vaibhav Srivastav and Joshua Lochner and Hugo Larcher and Mathieu Morlon and Lewis Tunstall and Leandro von Werra and Thomas Wolf},217 journal={arXiv preprint arXiv:2504.05299},218 year={2025}219}220```221 222## Training Data223SmolVLM2 used 3.3M samples for training originally from ten different datasets: [LlaVa Onevision](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Data), [M4-Instruct](https://huggingface.co/datasets/lmms-lab/M4-Instruct-Data), [Mammoth](https://huggingface.co/datasets/MAmmoTH-VL/MAmmoTH-VL-Instruct-12M), [LlaVa Video 178K](https://huggingface.co/datasets/lmms-lab/LLaVA-Video-178K), [FineVideo](https://huggingface.co/datasets/HuggingFaceFV/finevideo), [VideoStar](https://huggingface.co/datasets/orrzohar/Video-STaR), [VRipt](https://huggingface.co/datasets/Mutonix/Vript), [Vista-400K](https://huggingface.co/datasets/TIGER-Lab/VISTA-400K), [MovieChat](https://huggingface.co/datasets/Enxin/MovieChat-1K_train) and [ShareGPT4Video](https://huggingface.co/datasets/ShareGPT4Video/ShareGPT4Video).224In the following plots we give a general overview of the samples across modalities and the source of those samples.225<!--226<center><img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/smolvlm2_data_split.png" width="auto" height="auto" alt="Image description">227</center>228 229### Details230<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/smolvlm2_datadetails.png" width="auto" height="auto" alt="Image description"> -->231 232## Data Split per modality233 234| Data Type | Percentage |235|--------------|------------|236| Image | 34.4% |237| Text | 20.2% |238| Video | 33.0% |239| Multi-image | 12.3% |240 241 242## Granular dataset slices per modality243 244### Text Datasets245| Dataset | Percentage |246|--------------------------------------------|------------|247| llava-onevision/magpie_pro_ft3_80b_mt | 6.8% |248| llava-onevision/magpie_pro_ft3_80b_tt | 6.8% |249| llava-onevision/magpie_pro_qwen2_72b_tt | 5.8% |250| llava-onevision/mathqa | 0.9% |251 252### Multi-image Datasets253| Dataset | Percentage |254|--------------------------------------------|------------|255| m4-instruct-data/m4_instruct_multiimage | 10.4% |256| mammoth/multiimage-cap6 | 1.9% |257 258### Image Datasets259| Dataset | Percentage |260|--------------------------------------------|------------|261| llava-onevision/other | 17.4% |262| llava-onevision/vision_flan | 3.9% |263| llava-onevision/mavis_math_metagen | 2.6% |264| llava-onevision/mavis_math_rule_geo | 2.5% |265| llava-onevision/sharegpt4o | 1.7% |266| llava-onevision/sharegpt4v_coco | 1.5% |267| llava-onevision/image_textualization | 1.3% |268| llava-onevision/sharegpt4v_llava | 0.9% |269| llava-onevision/mapqa | 0.9% |270| llava-onevision/qa | 0.8% |271| llava-onevision/textocr | 0.8% |272 273### Video Datasets274| Dataset | Percentage |275|--------------------------------------------|------------|276| llava-video-178k/1-2m | 7.3% |277| llava-video-178k/2-3m | 7.0% |278| other-video/combined | 5.7% |279| llava-video-178k/hound | 4.4% |280| llava-video-178k/0-30s | 2.4% |281| video-star/starb | 2.2% |282| vista-400k/combined | 2.2% |283| vript/long | 1.0% |284| ShareGPT4Video/all | 0.8% |285 286