HuggingFaceTB/SmolVLM2-256M-Video-Instruct

1---2library_name: transformers3license: apache-2.04datasets:5- HuggingFaceM4/the_cauldron6- HuggingFaceM4/Docmatix7- lmms-lab/LLaVA-OneVision-Data8- lmms-lab/M4-Instruct-Data9- HuggingFaceFV/finevideo10- MAmmoTH-VL/MAmmoTH-VL-Instruct-12M11- lmms-lab/LLaVA-Video-178K12- orrzohar/Video-STaR13- Mutonix/Vript14- TIGER-Lab/VISTA-400K15- Enxin/MovieChat-1K_train16- ShareGPT4Video/ShareGPT4Video17pipeline_tag: image-text-to-text18language:19- en20base_model:21- HuggingFaceTB/SmolVLM-256M-Instruct22---23 24<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/SmolVLM2_banner.png" width="800" height="auto" alt="Image description">25 26# SmolVLM2-256M-Video27 28SmolVLM2-256M-Video is a lightweight multimodal model designed to analyze video content. The model processes videos, images, and text inputs to generate text outputs - whether answering questions about media files, comparing visual content, or transcribing text from images. Despite its compact size, requiring only 1.38GB of GPU RAM for video inference. This efficiency makes it particularly well-suited for on-device applications that require specific domain fine-tuning and computational resources may be limited.29## Model Summary30 31- **Developed by:** Hugging Face 🤗32- **Model type:** Multi-modal model (image/multi-image/video/text)33- **Language(s) (NLP):** English34- **License:** Apache 2.035- **Architecture:** Based on [Idefics3](https://huggingface.co/HuggingFaceM4/Idefics3-8B-Llama3) (see technical summary)36 37## Resources38 39- **Demo:** [Video Highlight Generator](https://huggingface.co/spaces/HuggingFaceTB/SmolVLM2-HighlightGenerator)40- **Blog:** [Blog post](https://huggingface.co/blog/smolvlm2)41 42## Uses43 44SmolVLM2 can be used for inference on multimodal (video / image / text) tasks where the input consists of text queries along with video or one or more images. Text and media files can be interleaved arbitrarily, enabling tasks like captioning, visual question answering, and storytelling based on visual content. The model does not support image or video generation.45 46To fine-tune SmolVLM2 on a specific task, you can follow [the fine-tuning tutorial](https://github.com/huggingface/smollm/blob/main/vision/finetuning/Smol_VLM_FT.ipynb).47 48## Evaluation 49 50We evaluated the performance of the SmolVLM2 family on the following scientific benchmarks:51 52| Size    | Video-MME | MLVU | MVBench |53|----------|-----------------|----------|---------------|54| 2.2B   | 52.1            | 55.2     | 46.27        |55| 500M | 42.2            | 47.3     | 39.73        |56| 256M | 33.7            | 40.6     | 32.7          |57 58 59### How to get started60 61You can use transformers to load, infer and fine-tune SmolVLM. Make sure you have num2words, flash-attn and latest transformers installed.62You can load the model as follows.63 64```python65from transformers import AutoProcessor, AutoModelForImageTextToText66import torch67 68model_path = "HuggingFaceTB/SmolVLM2-256M-Video-Instruct"69processor = AutoProcessor.from_pretrained(model_path)70model = AutoModelForImageTextToText.from_pretrained(71    model_path,72    torch_dtype=torch.bfloat16,73    _attn_implementation="flash_attention_2"74).to("cuda")75```76 77#### Simple Inference78 79You preprocess your inputs directly using chat templates and directly passing them 80 81```python82messages = [83    {84        "role": "user",85        "content": [86            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"},87            {"type": "text", "text": "Can you describe this image?"},            88        ]89    },90]91 92inputs = processor.apply_chat_template(93    messages,94    add_generation_prompt=True,95    tokenize=True,96    return_dict=True,97    return_tensors="pt",98).to(model.device, dtype=torch.bfloat16)99 100generated_ids = model.generate(**inputs, do_sample=False, max_new_tokens=64)101generated_texts = processor.batch_decode(102    generated_ids,103    skip_special_tokens=True,104)105print(generated_texts[0])106```107 108#### Video Inference109 110To use SmolVLM2 for video inference, make sure you have decord installed. 111 112```python113messages = [114    {115        "role": "user",116        "content": [117            {"type": "video", "path": "path_to_video.mp4"},118            {"type": "text", "text": "Describe this video in detail"}119        ]120    },121]122 123inputs = processor.apply_chat_template(124    messages,125    add_generation_prompt=True,126    tokenize=True,127    return_dict=True,128    return_tensors="pt",129).to(model.device, dtype=torch.bfloat16)130 131generated_ids = model.generate(**inputs, do_sample=False, max_new_tokens=64)132generated_texts = processor.batch_decode(133    generated_ids,134    skip_special_tokens=True,135)136 137print(generated_texts[0])138```139#### Multi-image Interleaved Inference140 141You can interleave multiple media with text using chat templates.142 143```python144import torch145 146 147messages = [148    {149        "role": "user",150        "content": [151          {"type": "text", "text": "What is the similarity between these two images?"},152          {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"},153          {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"},            154        ]155    },156]157inputs = processor.apply_chat_template(158    messages,159    add_generation_prompt=True,160    tokenize=True,161    return_dict=True,162    return_tensors="pt",163).to(model.device, dtype=torch.bfloat16)164 165generated_ids = model.generate(**inputs, do_sample=False, max_new_tokens=64)166generated_texts = processor.batch_decode(167    generated_ids,168    skip_special_tokens=True,169)170print(generated_texts[0])171```172 173 174### Model optimizations175 176## Misuse and Out-of-scope Use177 178SmolVLM is not intended for high-stakes scenarios or critical decision-making processes that affect an individual's well-being or livelihood. The model may produce content that appears factual but may not be accurate. Misuse includes, but is not limited to:179 180- Prohibited Uses:181  - Evaluating or scoring individuals (e.g., in employment, education, credit)182  - Critical automated decision-making183  - Generating unreliable factual content184- Malicious Activities:185  - Spam generation186  - Disinformation campaigns187  - Harassment or abuse188  - Unauthorized surveillance189 190### License191 192SmolVLM2 is built upon [SigLIP](https://huggingface.co/google/siglip-base-patch16-512) as image encoder and [SmolLM2](https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct) for text decoder part.193 194We release the SmolVLM2 checkpoints under the Apache 2.0 license.195 196## Citation information197You can cite us in the following way:198```bibtex199@article{marafioti2025smolvlm,200  title={SmolVLM: Redefining small and efficient multimodal models}, 201  author={Andrés Marafioti and Orr Zohar and Miquel Farré and Merve Noyan and Elie Bakouch and Pedro Cuenca and Cyril Zakka and Loubna Ben Allal and Anton Lozhkov and Nouamane Tazi and Vaibhav Srivastav and Joshua Lochner and Hugo Larcher and Mathieu Morlon and Lewis Tunstall and Leandro von Werra and Thomas Wolf},202  journal={arXiv preprint arXiv:2504.05299},203  year={2025}204}205```206 207## Training Data208SmolVLM2 used 3.3M samples for training originally from ten different datasets: [LlaVa Onevision](https://huggingface.co/datasets/lmms-lab/LLaVA-OneVision-Data), [M4-Instruct](https://huggingface.co/datasets/lmms-lab/M4-Instruct-Data), [Mammoth](https://huggingface.co/datasets/MAmmoTH-VL/MAmmoTH-VL-Instruct-12M), [LlaVa Video 178K](https://huggingface.co/datasets/lmms-lab/LLaVA-Video-178K), [FineVideo](https://huggingface.co/datasets/HuggingFaceFV/finevideo), [VideoStar](https://huggingface.co/datasets/orrzohar/Video-STaR), [VRipt](https://huggingface.co/datasets/Mutonix/Vript), [Vista-400K](https://huggingface.co/datasets/TIGER-Lab/VISTA-400K), [MovieChat](https://huggingface.co/datasets/Enxin/MovieChat-1K_train) and [ShareGPT4Video](https://huggingface.co/datasets/ShareGPT4Video/ShareGPT4Video).209In the following plots we give a general overview of the samples across modalities and the source of those samples.210<!--211<center><img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/smolvlm2_data_split.png" width="auto" height="auto" alt="Image description">212</center>213 214### Details215<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/smolvlm2_datadetails.png" width="auto" height="auto" alt="Image description"> -->216 217## Data Split per modality218 219| Data Type    | Percentage |220|--------------|------------|221| Image        | 34.4%      |222| Text         | 20.2%      |223| Video        | 33.0%      |224| Multi-image  | 12.3%      |225 226 227## Granular dataset slices per modality228 229### Text Datasets230| Dataset                                    | Percentage |231|--------------------------------------------|------------|232| llava-onevision/magpie_pro_ft3_80b_mt      | 6.8%       |233| llava-onevision/magpie_pro_ft3_80b_tt      | 6.8%       |234| llava-onevision/magpie_pro_qwen2_72b_tt    | 5.8%       |235| llava-onevision/mathqa                     | 0.9%       |236 237### Multi-image Datasets238| Dataset                                    | Percentage |239|--------------------------------------------|------------|240| m4-instruct-data/m4_instruct_multiimage    | 10.4%      |241| mammoth/multiimage-cap6                    | 1.9%       |242 243### Image Datasets244| Dataset                                    | Percentage |245|--------------------------------------------|------------|246| llava-onevision/other                      | 17.4%      |247| llava-onevision/vision_flan                | 3.9%       |248| llava-onevision/mavis_math_metagen         | 2.6%       |249| llava-onevision/mavis_math_rule_geo        | 2.5%       |250| llava-onevision/sharegpt4o                 | 1.7%       |251| llava-onevision/sharegpt4v_coco            | 1.5%       |252| llava-onevision/image_textualization       | 1.3%       |253| llava-onevision/sharegpt4v_llava           | 0.9%       |254| llava-onevision/mapqa                      | 0.9%       |255| llava-onevision/qa                         | 0.8%       |256| llava-onevision/textocr                    | 0.8%       |257 258### Video Datasets259| Dataset                                    | Percentage |260|--------------------------------------------|------------|261| llava-video-178k/1-2m                      | 7.3%       |262| llava-video-178k/2-3m                      | 7.0%       |263| other-video/combined                       | 5.7%       |264| llava-video-178k/hound                     | 4.4%       |265| llava-video-178k/0-30s                     | 2.4%       |266| video-star/starb                           | 2.2%       |267| vista-400k/combined                        | 2.2%       |268| vript/long                                 | 1.0%       |269| ShareGPT4Video/all                         | 0.8%       |270