M-CLIP/XLM-Roberta-Large-Vit-B-32
Multilingual-clip: XLM-Roberta-Large-Vit-B-32
Multilingual-CLIP extends OpenAI's English text encoders to multiple other languages. This model only contains the multilingual text encoder. The corresponding image model ViT-B-32 can be retrieved via instructions found on OpenAI's CLIP repository on Github. We provide a usage example below.
Requirements
To use both the multilingual text encoder and corresponding image encoder, we need to install the packages `multilingual-clip` and `clip`.
pip install multilingual-clip
pip install git+https://github.com/openai/CLIP.gitUsage
Extracting embeddings from the text encoder can be done in the following way:
from multilingual_clip import pt_multilingual_clip
import transformers
texts = [
'Three blind horses listening to Mozart.',
'Älgen är skogens konung!',
'Wie leben Eisbären in der Antarktis?',
'Вы знали, что все белые медведи левши?'
]
model_name = 'M-CLIP/XLM-Roberta-Large-Vit-B-32'
# Load Model & Tokenizer
model = pt_multilingual_clip.MultilingualCLIP.from_pretrained(model_name)
tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)
embeddings = model.forward(texts, tokenizer)
print("Text features shape:", embeddings.shape)Extracting embeddings from the corresponding image encoder:
import torch
import clip
import requests
from PIL import Image
device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
image = preprocess(image).unsqueeze(0).to(device)
with torch.no_grad():
image_features = model.encode_image(image)
print("Image features shape:", image_features.shape) Evaluation results
None of the M-CLIP models have been extensivly evaluated, but testing them on Txt2Img retrieval on the humanly translated MS-COCO dataset, we see the following R@10 results:
Training/Model details
Further details about the model training and data can be found in the model card.