TIGER-Lab/VLM2Vec-Full

1---2license: apache-2.03datasets:4- TIGER-Lab/MMEB-train5language:6- en7metrics:8- accuracy9base_model:10- microsoft/Phi-3.5-vision-instruct11library_name: transformers12tags:13- Embedding14---15 16# VLM2Vec17 18This repo contains the code and data for [VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks](https://arxiv.org/abs/2410.05160). In this paper, we aimed at building a unified multimodal embedding model for any tasks. Our model is based on converting an existing well-trained VLM (Phi-3.5-V) into an embedding model.19 20<img width="1432" alt="abs" src="https://raw.githubusercontent.com/TIGER-AI-Lab/VLM2Vec/refs/heads/main/figures//train_vlm.png">21 22**We’ve released several VLM2Vec models built on different VLM backbones: https://huggingface.co/collections/TIGER-Lab/vlm2vec-6705f418271d085836e0cdd5**23 24**Also, the performance of these models is updated in the README of our GitHub repository: https://github.com/TIGER-AI-Lab/VLM2Vec/blob/main/README.md**25 26## Release27Our model is being trained on MMEB-train and evaluated on MMEB-eval with contrastive learning. We only use in-batch negatives for training. Our best results were based on Lora training with batch size of 1024. We also have checkpoint with full training with batch size of 2048. Our results on 36 evaluation datasets are:28### Train/Eval Data29 - Train data: https://huggingface.co/datasets/TIGER-Lab/MMEB-train30 - Eval data: https://huggingface.co/datasets/TIGER-Lab/MMEB-eval31 32### VLM2Vec Checkpoints33 - [MMEB.lora8.bs1024](https://huggingface.co/TIGER-Lab/MMEB.lora8.bs1024/)34 - [MMEB.fullmodel.bs2048](https://huggingface.co/TIGER-Lab/MMEB.fullmodel.bs2048/)35 36### Github37 - [Github](https://github.com/TIGER-AI-Lab/VLM2Vec)38 39### Experimental Results40Our model can outperform the existing baselines by a huge margin.41<img width="900" alt="abs" src="https://raw.githubusercontent.com/TIGER-AI-Lab/VLM2Vec/refs/heads/main/figures//vlm2vec_results.png">42 43## How to use VLM2Vec44 45First you can clone our github46```bash47git clone https://github.com/TIGER-AI-Lab/VLM2Vec.git48pip -r requirements.txt49```50 51Then you can enter the directory to run the following command.52```python53from src.model import MMEBModel54from src.arguments import ModelArguments55from src.utils import load_processor56import torch57from transformers import HfArgumentParser, AutoProcessor58from PIL import Image59import numpy as np60 61model_args = ModelArguments(62    model_name='TIGER-Lab/VLM2Vec-Full',63    pooling='last',64    normalize=True,65    model_backbone='phi3_v',66    num_crops=16)67 68processor = load_processor(model_args)69 70model = MMEBModel.load(model_args)71model.eval()72model = model.to('cuda', dtype=torch.bfloat16)73 74 75# Image + Text -> Text76inputs = processor('<|image_1|> Represent the given image with the following question: What is in the image', [Image.open(77    'figures/example.jpg')])78inputs = {key: value.to('cuda') for key, value in inputs.items()}79qry_output = model(qry=inputs)["qry_reps"]80 81string = 'A cat and a dog'82inputs = processor(string)83inputs = {key: value.to('cuda') for key, value in inputs.items()}84tgt_output = model(tgt=inputs)["tgt_reps"]85print(string, '=', model.compute_similarity(qry_output, tgt_output))86## A cat and a dog = tensor([[0.3008]], device='cuda:0', dtype=torch.bfloat16)87 88string = 'A cat and a tiger'89inputs = processor(string)90inputs = {key: value.to('cuda') for key, value in inputs.items()}91tgt_output = model(tgt=inputs)["tgt_reps"]92print(string, '=', model.compute_similarity(qry_output, tgt_output))93## A cat and a tiger = tensor([[0.2051]], device='cuda:0', dtype=torch.bfloat16)94 95# Text -> Image96inputs = processor('Find me an everyday image that matches the given caption: A cat and a dog.',)97inputs = {key: value.to('cuda') for key, value in inputs.items()}98qry_output = model(qry=inputs)["qry_reps"]99 100string = '<|image_1|> Represent the given image.'101inputs = processor(string, [Image.open('figures/example.jpg')])102inputs = {key: value.to('cuda') for key, value in inputs.items()}103tgt_output = model(tgt=inputs)["tgt_reps"]104print(string, '=', model.compute_similarity(qry_output, tgt_output))105## <|image_1|> Represent the given image. = tensor([[0.2930]], device='cuda:0', dtype=torch.bfloat16)106 107inputs = processor('Find me an everyday image that matches the given caption: A cat and a tiger.',)108inputs = {key: value.to('cuda') for key, value in inputs.items()}109qry_output = model(qry=inputs)["qry_reps"]110 111string = '<|image_1|> Represent the given image.'112inputs = processor(string, [Image.open('figures/example.jpg')])113inputs = {key: value.to('cuda') for key, value in inputs.items()}114tgt_output = model(tgt=inputs)["tgt_reps"]115print(string, '=', model.compute_similarity(qry_output, tgt_output))116## <|image_1|> Represent the given image. = tensor([[0.2012]], device='cuda:0', dtype=torch.bfloat16)117```118 119## Citation120```121@article{jiang2024vlm2vec,122  title={VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks},123  author={Jiang, Ziyan and Meng, Rui and Yang, Xinyi and Yavuz, Semih and Zhou, Yingbo and Chen, Wenhu},124  journal={arXiv preprint arXiv:2410.05160},125  year={2024}126}