OEvortex/HelpingAI-Vision

1---2language:3- en4license: other5license_name: hsul6license_link: https://huggingface.co/OEvortex/vortex-3b/raw/main/LICENSE.md7library_name: transformers8base_model: visheratin/MC-LLaVA-3b9widget:10- text: What animal is it?11  src: https://huggingface.co/datasets/mishig/sample_images/resolve/main/tiger.jpg12- text: Where is it?13  src: https://huggingface.co/datasets/mishig/sample_images/resolve/main/palace.jpg14pipeline_tag: image-text-to-text15---16 17# HelpingAI-Vision18 19<a target="_blank" href="https://colab.research.google.com/drive/1t2OAMVSKsiqVgvuHq7rhyNv28b67u0D8">20  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>21</a>22 23## Model details24 25The fundamental concept behind HelpingAI-Vision is to generate one token embedding per N parts of an image, as opposed to producing N visual token embeddings for the entire image. This approach, based on the HelpingAI-Lite and incorporating the LLaVA adapter, aims to enhance scene understanding by capturing more detailed information.26 27For every crop of the image, an embedding is generated using the full SigLIP encoder (size [1, 1152]). Subsequently, all N embeddings undergo processing through the LLaVA adapter, resulting in a token embedding of size [N, 2560]. Currently, these tokens lack explicit information about their position in the original image, with plans to incorporate positional information in a later update.28 29HelpingAI-Vision was fine-tuned from MC-LLaVA-3b.30 31The model adopts the ChatML prompt format, suggesting its potential application in chat-based scenarios. If you have specific queries or would like further details, feel free ask32```33<|im_start|>system34You are Vortex, a helpful AI assistant.<|im_end|>35<|im_start|>user36{prompt}<|im_end|>37<|im_start|>assistant38```39 40## How to use41 42**Install dependencies**43 44```bash45!pip install -q open_clip_torch timm einops46```47 48**Download modeling files**49 50```python51from huggingface_hub import hf_hub_download52 53hf_hub_download(repo_id="OEvortex/HelpingAI-Vision", filename="configuration_llava.py", local_dir="./", force_download=True)54hf_hub_download(repo_id="OEvortex/HelpingAI-Vision", filename="configuration_phi.py", local_dir="./", force_download=True)55hf_hub_download(repo_id="OEvortex/HelpingAI-Vision", filename="modeling_llava.py", local_dir="./", force_download=True)56hf_hub_download(repo_id="OEvortex/HelpingAI-Vision", filename="modeling_phi.py", local_dir="./", force_download=True)57hf_hub_download(repo_id="OEvortex/HelpingAI-Vision", filename="processing_llava.py", local_dir="./", force_download=True)58```59 60**Create a model**61 62```python63from modeling_llava import LlavaForConditionalGeneration64import torch65 66model = LlavaForConditionalGeneration.from_pretrained("OEvortex/HelpingAI-Vision", torch_dtype=torch.float16)67model = model.to("cuda")68```69 70**Create processors**71 72```python73from transformers import AutoTokenizer74from processing_llava import LlavaProcessor, OpenCLIPImageProcessor75 76tokenizer = AutoTokenizer.from_pretrained("OEvortex/HelpingAI-Vision")77image_processor = OpenCLIPImageProcessor(model.config.preprocess_config)78processor = LlavaProcessor(image_processor, tokenizer)79```80 81**Set image and text**82 83```python84from PIL import Image85import requests86 87image_file = "https://images.unsplash.com/photo-1439246854758-f686a415d9da"88raw_image = Image.open(requests.get(image_file, stream=True).raw)89 90prompt = """<|im_start|>system91A chat between a curious human and an artificial intelligence assistant.92The assistant gives helpful, detailed, and polite answers to the human's questions.93The assistant does not hallucinate and pays very close attention to the details.<|im_end|>94<|im_start|>user95<image>96Describe the image.<|im_end|>97<|im_start|>assistant98"""99```100 101**Process inputs**102 103```python104with torch.inference_mode():105  inputs = processor(prompt, raw_image, model, return_tensors='pt')106 107inputs['input_ids'] = inputs['input_ids'].to(model.device)108inputs['attention_mask'] = inputs['attention_mask'].to(model.device)109 110from transformers import TextStreamer111 112streamer = TextStreamer(tokenizer)113```114 115**Generate the data**116 117```python118%%time119with torch.inference_mode():120  output = model.generate(**inputs, max_new_tokens=200, do_sample=True, top_p=0.9, temperature=1.2, eos_token_id=tokenizer.eos_token_id, streamer=streamer)121print(tokenizer.decode(output[0]).replace(prompt, "").replace("<|im_end|>", ""))122```