OpenGVLab/InternVL2-40B
93285
1---2license: mit3pipeline_tag: image-text-to-text4library_name: transformers5base_model:6 - OpenGVLab/InternViT-6B-448px-V1-57 - NousResearch/Nous-Hermes-2-Yi-34B8new_version: OpenGVLab/InternVL2_5-38B9base_model_relation: merge10language:11 - multilingual12tags:13 - internvl14 - custom_code15---16 17# InternVL2-40B18 19[\[📂 GitHub\]](https://github.com/OpenGVLab/InternVL) [\[📜 InternVL 1.0\]](https://huggingface.co/papers/2312.14238) [\[📜 InternVL 1.5\]](https://huggingface.co/papers/2404.16821) [\[📜 Mini-InternVL\]](https://arxiv.org/abs/2410.16261) [\[📜 InternVL 2.5\]](https://huggingface.co/papers/2412.05271)20 21[\[🆕 Blog\]](https://internvl.github.io/blog/) [\[🗨️ Chat Demo\]](https://internvl.opengvlab.com/) [\[🤗 HF Demo\]](https://huggingface.co/spaces/OpenGVLab/InternVL) [\[🚀 Quick Start\]](#quick-start) [\[📖 Documents\]](https://internvl.readthedocs.io/en/latest/)22 23<div align="center">24 <img width="500" alt="image" src="https://cdn-uploads.huggingface.co/production/uploads/64006c09330a45b03605bba3/zJsd2hqd3EevgXo6fNgC-.png">25</div>26 27## Introduction28 29We are excited to announce the release of InternVL 2.0, the latest addition to the InternVL series of multimodal large language models. InternVL 2.0 features a variety of **instruction-tuned models**, ranging from 1 billion to 108 billion parameters. This repository contains the instruction-tuned InternVL2-40B model.30 31Compared to the state-of-the-art open-source multimodal large language models, InternVL 2.0 surpasses most open-source models. It demonstrates competitive performance on par with proprietary commercial models across various capabilities, including document and chart comprehension, infographics QA, scene text understanding and OCR tasks, scientific and mathematical problem solving, as well as cultural understanding and integrated multimodal capabilities.32 33InternVL 2.0 is trained with an 8k context window and utilizes training data consisting of long texts, multiple images, and videos, significantly improving its ability to handle these types of inputs compared to InternVL 1.5. For more details, please refer to our [blog](https://internvl.github.io/blog/2024-07-02-InternVL-2.0/) and [GitHub](https://github.com/OpenGVLab/InternVL).34 35| Model Name | Vision Part | Language Part | HF Link | MS Link |36| :------------------: | :---------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------: | :--------------------------------------------------------------: | :--------------------------------------------------------------------: |37| InternVL2-1B | [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px) | [Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-1B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-1B) |38| InternVL2-2B | [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px) | [internlm2-chat-1_8b](https://huggingface.co/internlm/internlm2-chat-1_8b) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-2B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-2B) |39| InternVL2-4B | [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px) | [Phi-3-mini-128k-instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-4B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-4B) |40| InternVL2-8B | [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px) | [internlm2_5-7b-chat](https://huggingface.co/internlm/internlm2_5-7b-chat) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-8B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-8B) |41| InternVL2-26B | [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5) | [internlm2-chat-20b](https://huggingface.co/internlm/internlm2-chat-20b) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-26B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-26B) |42| InternVL2-40B | [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5) | [Nous-Hermes-2-Yi-34B](https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-40B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-40B) |43| InternVL2-Llama3-76B | [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5) | [Hermes-2-Theta-Llama-3-70B](https://huggingface.co/NousResearch/Hermes-2-Theta-Llama-3-70B) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2-Llama3-76B) | [🤖 link](https://modelscope.cn/models/OpenGVLab/InternVL2-Llama3-76B) |44 45## Model Details46 47InternVL 2.0 is a multimodal large language model series, featuring models of various sizes. For each size, we release instruction-tuned models optimized for multimodal tasks. InternVL2-40B consists of [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5), an MLP projector, and [Nous-Hermes-2-Yi-34B](https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B).48 49## Performance50 51### Image Benchmarks52 53| Benchmark | GPT-4T-20240409 | Gemini-1.5-Pro | InternVL2-26B | InternVL2-40B |54| :--------------------------: | :-------------: | :------------: | :-----------: | :-----------: |55| Model Size | - | - | 25.5B | 40B |56| | | | | |57| DocVQA<sub>test</sub> | 87.2 | 86.5 | 92.9 | 93.9 |58| ChartQA<sub>test</sub> | 78.1 | 81.3 | 84.9 | 86.2 |59| InfoVQA<sub>test</sub> | - | 72.7 | 75.9 | 78.7 |60| TextVQA<sub>val</sub> | - | 73.5 | 82.3 | 83.0 |61| OCRBench | 678 | 754 | 825 | 837 |62| MME<sub>sum</sub> | 2070.2 | 2110.6 | 2260.7 | 2315.0 |63| RealWorldQA | 68.0 | 67.5 | 68.3 | 71.8 |64| AI2D<sub>test</sub> | 89.4 | 80.3 | 84.5 | 87.1 |65| MMMU<sub>val</sub> | 63.1 | 58.5 | 51.2 | 55.2 |66| MMBench-EN<sub>test</sub> | 81.0 | 73.9 | 83.4 | 86.8 |67| MMBench-CN<sub>test</sub> | 80.2 | 73.8 | 82.0 | 86.5 |68| CCBench<sub>dev</sub> | 57.3 | 28.4 | 73.5 | 80.6 |69| MMVet<sub>GPT-4-0613</sub> | - | - | 64.2 | 68.5 |70| MMVet<sub>GPT-4-Turbo</sub> | 67.5 | 64.0 | 62.1 | 65.5 |71| SEED-Image | - | - | 76.8 | 78.2 |72| HallBench<sub>avg</sub> | 43.9 | 45.6 | 50.7 | 56.9 |73| MathVista<sub>testmini</sub> | 58.1 | 57.7 | 59.4 | 63.7 |74| OpenCompass<sub>avg</sub> | 63.5 | 64.4 | 66.4 | 69.7 |75 76- For more details and evaluation reproduction, please refer to our [Evaluation Guide](https://internvl.readthedocs.io/en/latest/internvl2.0/evaluation.html).77 78- We simultaneously use [InternVL](https://github.com/OpenGVLab/InternVL) and [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) repositories for model evaluation. Specifically, the results reported for DocVQA, ChartQA, InfoVQA, TextVQA, MME, AI2D, MMBench, CCBench, MMVet (GPT-4-0613), and SEED-Image were tested using the InternVL repository. MMMU, OCRBench, RealWorldQA, HallBench, MMVet (GPT-4-Turbo), and MathVista were evaluated using the VLMEvalKit.79 80### Video Benchmarks81 82| Benchmark | GPT-4V | VILA-1.5 | LLaVA-NeXT-Video | InternVL2-26B | InternVL2-40B |83| :-------------------------: | :----: | :------: | :--------------: | :-----------: | :-----------: |84| Model Size | - | 34B | 34B | 25.5B | 40B |85| | | | | | |86| MVBench | - | - | - | 67.5 | 72.5 |87| MMBench-Video<sub>8f</sub> | 1.53 | - | - | 1.27 | 1.32 |88| MMBench-Video<sub>16f</sub> | 1.68 | - | - | 1.41 | 1.45 |89| Video-MME<br>w/o subs | 59.9 | 59.0 | 52.0 | 54.8 | 61.2 |90| Video-MME<br>w subs | 63.3 | 59.4 | 54.9 | 57.1 | 62.4 |91 92- We evaluate our models on MVBench and Video-MME by extracting 16 frames from each video, and each frame was resized to a 448x448 image.93 94### Grounding Benchmarks95 96| Model | avg. | RefCOCO<br>(val) | RefCOCO<br>(testA) | RefCOCO<br>(testB) | RefCOCO+<br>(val) | RefCOCO+<br>(testA) | RefCOCO+<br>(testB) | RefCOCO‑g<br>(val) | RefCOCO‑g<br>(test) |97| :----------------------------: | :--: | :--------------: | :----------------: | :----------------: | :---------------: | :-----------------: | :-----------------: | :----------------: | :-----------------: |98| UNINEXT-H<br>(Specialist SOTA) | 88.9 | 92.6 | 94.3 | 91.5 | 85.2 | 89.6 | 79.8 | 88.7 | 89.4 |99| | | | | | | | | | |100| Mini-InternVL-<br>Chat-2B-V1-5 | 75.8 | 80.7 | 86.7 | 72.9 | 72.5 | 82.3 | 60.8 | 75.6 | 74.9 |101| Mini-InternVL-<br>Chat-4B-V1-5 | 84.4 | 88.0 | 91.4 | 83.5 | 81.5 | 87.4 | 73.8 | 84.7 | 84.6 |102| InternVL‑Chat‑V1‑5 | 88.8 | 91.4 | 93.7 | 87.1 | 87.0 | 92.3 | 80.9 | 88.5 | 89.3 |103| | | | | | | | | | |104| InternVL2‑1B | 79.9 | 83.6 | 88.7 | 79.8 | 76.0 | 83.6 | 67.7 | 80.2 | 79.9 |105| InternVL2‑2B | 77.7 | 82.3 | 88.2 | 75.9 | 73.5 | 82.8 | 63.3 | 77.6 | 78.3 |106| InternVL2‑4B | 84.4 | 88.5 | 91.2 | 83.9 | 81.2 | 87.2 | 73.8 | 84.6 | 84.6 |107| InternVL2‑8B | 82.9 | 87.1 | 91.1 | 80.7 | 79.8 | 87.9 | 71.4 | 82.7 | 82.7 |108| InternVL2‑26B | 88.5 | 91.2 | 93.3 | 87.4 | 86.8 | 91.0 | 81.2 | 88.5 | 88.6 |109| InternVL2‑40B | 90.3 | 93.0 | 94.7 | 89.2 | 88.5 | 92.8 | 83.6 | 90.3 | 90.6 |110| InternVL2-<br>Llama3‑76B | 90.0 | 92.2 | 94.8 | 88.4 | 88.8 | 93.1 | 82.8 | 89.5 | 90.3 |111 112- We use the following prompt to evaluate InternVL's grounding ability: `Please provide the bounding box coordinates of the region this sentence describes: <ref>{}</ref>`113 114Limitations: Although we have made efforts to ensure the safety of the model during the training process and to encourage the model to generate text that complies with ethical and legal requirements, the model may still produce unexpected outputs due to its size and probabilistic generation paradigm. For example, the generated responses may contain biases, discrimination, or other harmful content. Please do not propagate such content. We are not responsible for any consequences resulting from the dissemination of harmful information.115 116## Quick Start117 118We provide an example code to run `InternVL2-40B` using `transformers`.119 120> Please use transformers>=4.37.2 to ensure the model works normally.121 122### Model Loading123 124#### 16-bit (bf16 / fp16)125 126```python127import torch128from transformers import AutoTokenizer, AutoModel129path = "OpenGVLab/InternVL2-40B"130model = AutoModel.from_pretrained(131 path,132 torch_dtype=torch.bfloat16,133 low_cpu_mem_usage=True,134 use_flash_attn=True,135 trust_remote_code=True).eval().cuda()136```137 138#### BNB 8-bit Quantization139 140```python141import torch142from transformers import AutoTokenizer, AutoModel143path = "OpenGVLab/InternVL2-40B"144model = AutoModel.from_pretrained(145 path,146 torch_dtype=torch.bfloat16,147 load_in_8bit=True,148 low_cpu_mem_usage=True,149 use_flash_attn=True,150 trust_remote_code=True).eval()151```152 153#### Multiple GPUs154 155The reason for writing the code this way is to avoid errors that occur during multi-GPU inference due to tensors not being on the same device. By ensuring that the first and last layers of the large language model (LLM) are on the same device, we prevent such errors.156 157```python158import math159import torch160from transformers import AutoTokenizer, AutoModel161 162def split_model(model_name):163 device_map = {}164 world_size = torch.cuda.device_count()165 num_layers = {166 'InternVL2-1B': 24, 'InternVL2-2B': 24, 'InternVL2-4B': 32, 'InternVL2-8B': 32,167 'InternVL2-26B': 48, 'InternVL2-40B': 60, 'InternVL2-Llama3-76B': 80}[model_name]168 # Since the first GPU will be used for ViT, treat it as half a GPU.169 num_layers_per_gpu = math.ceil(num_layers / (world_size - 0.5))170 num_layers_per_gpu = [num_layers_per_gpu] * world_size171 num_layers_per_gpu[0] = math.ceil(num_layers_per_gpu[0] * 0.5)172 layer_cnt = 0173 for i, num_layer in enumerate(num_layers_per_gpu):174 for j in range(num_layer):175 device_map[f'language_model.model.layers.{layer_cnt}'] = i176 layer_cnt += 1177 device_map['vision_model'] = 0178 device_map['mlp1'] = 0179 device_map['language_model.model.tok_embeddings'] = 0180 device_map['language_model.model.embed_tokens'] = 0181 device_map['language_model.output'] = 0182 device_map['language_model.model.norm'] = 0183 device_map['language_model.model.rotary_emb'] = 0184 device_map['language_model.lm_head'] = 0185 device_map[f'language_model.model.layers.{num_layers - 1}'] = 0186 187 return device_map188 189path = "OpenGVLab/InternVL2-40B"190device_map = split_model('InternVL2-26B')191model = AutoModel.from_pretrained(192 path,193 torch_dtype=torch.bfloat16,194 low_cpu_mem_usage=True,195 use_flash_attn=True,196 trust_remote_code=True,197 device_map=device_map).eval()198```199 200### Inference with Transformers201 202```python203import math204import numpy as np205import torch206import torchvision.transforms as T207from decord import VideoReader, cpu208from PIL import Image209from torchvision.transforms.functional import InterpolationMode210from transformers import AutoModel, AutoTokenizer211 212IMAGENET_MEAN = (0.485, 0.456, 0.406)213IMAGENET_STD = (0.229, 0.224, 0.225)214 215def build_transform(input_size):216 MEAN, STD = IMAGENET_MEAN, IMAGENET_STD217 transform = T.Compose([218 T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img),219 T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),220 T.ToTensor(),221 T.Normalize(mean=MEAN, std=STD)222 ])223 return transform224 225def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):226 best_ratio_diff = float('inf')227 best_ratio = (1, 1)228 area = width * height229 for ratio in target_ratios:230 target_aspect_ratio = ratio[0] / ratio[1]231 ratio_diff = abs(aspect_ratio - target_aspect_ratio)232 if ratio_diff < best_ratio_diff:233 best_ratio_diff = ratio_diff234 best_ratio = ratio235 elif ratio_diff == best_ratio_diff:236 if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]:237 best_ratio = ratio238 return best_ratio239 240def dynamic_preprocess(image, min_num=1, max_num=12, image_size=448, use_thumbnail=False):241 orig_width, orig_height = image.size242 aspect_ratio = orig_width / orig_height243 244 # calculate the existing image aspect ratio245 target_ratios = set(246 (i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if247 i * j <= max_num and i * j >= min_num)248 target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])249 250 # find the closest aspect ratio to the target251 target_aspect_ratio = find_closest_aspect_ratio(252 aspect_ratio, target_ratios, orig_width, orig_height, image_size)253 254 # calculate the target width and height255 target_width = image_size * target_aspect_ratio[0]256 target_height = image_size * target_aspect_ratio[1]257 blocks = target_aspect_ratio[0] * target_aspect_ratio[1]258 259 # resize the image260 resized_img = image.resize((target_width, target_height))261 processed_images = []262 for i in range(blocks):263 box = (264 (i % (target_width // image_size)) * image_size,265 (i // (target_width // image_size)) * image_size,266 ((i % (target_width // image_size)) + 1) * image_size,267 ((i // (target_width // image_size)) + 1) * image_size268 )269 # split the image270 split_img = resized_img.crop(box)271 processed_images.append(split_img)272 assert len(processed_images) == blocks273 if use_thumbnail and len(processed_images) != 1:274 thumbnail_img = image.resize((image_size, image_size))275 processed_images.append(thumbnail_img)276 return processed_images277 278def load_image(image_file, input_size=448, max_num=12):279 image = Image.open(image_file).convert('RGB')280 transform = build_transform(input_size=input_size)281 images = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num)282 pixel_values = [transform(image) for image in images]283 pixel_values = torch.stack(pixel_values)284 return pixel_values285 286def split_model(model_name):287 device_map = {}288 world_size = torch.cuda.device_count()289 num_layers = {290 'InternVL2-1B': 24, 'InternVL2-2B': 24, 'InternVL2-4B': 32, 'InternVL2-8B': 32,291 'InternVL2-26B': 48, 'InternVL2-40B': 60, 'InternVL2-Llama3-76B': 80}[model_name]292 # Since the first GPU will be used for ViT, treat it as half a GPU.293 num_layers_per_gpu = math.ceil(num_layers / (world_size - 0.5))294 num_layers_per_gpu = [num_layers_per_gpu] * world_size295 num_layers_per_gpu[0] = math.ceil(num_layers_per_gpu[0] * 0.5)296 layer_cnt = 0297 for i, num_layer in enumerate(num_layers_per_gpu):298 for j in range(num_layer):299 device_map[f'language_model.model.layers.{layer_cnt}'] = i300 layer_cnt += 1301 device_map['vision_model'] = 0302 device_map['mlp1'] = 0303 device_map['language_model.model.tok_embeddings'] = 0304 device_map['language_model.model.embed_tokens'] = 0305 device_map['language_model.output'] = 0306 device_map['language_model.model.norm'] = 0307 device_map['language_model.model.rotary_emb'] = 0308 device_map['language_model.lm_head'] = 0309 device_map[f'language_model.model.layers.{num_layers - 1}'] = 0310 311 return device_map312 313# If you set `load_in_8bit=True`, you will need one 80GB GPUs.314# If you set `load_in_8bit=False`, you will need at least two 80GB GPUs.315path = 'OpenGVLab/InternVL2-40B'316device_map = split_model('InternVL2-40B')317model = AutoModel.from_pretrained(318 path,319 torch_dtype=torch.bfloat16,320 load_in_8bit=True,321 low_cpu_mem_usage=True,322 use_flash_attn=True,323 trust_remote_code=True,324 device_map=device_map).eval()325tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True, use_fast=False)326 327# set the max number of tiles in `max_num`328pixel_values = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()329generation_config = dict(max_new_tokens=1024, do_sample=True)330 331# pure-text conversation (纯文本对话)332question = 'Hello, who are you?'333response, history = model.chat(tokenizer, None, question, generation_config, history=None, return_history=True)334print(f'User: {question}\nAssistant: {response}')335 336question = 'Can you tell me a story?'337response, history = model.chat(tokenizer, None, question, generation_config, history=history, return_history=True)338print(f'User: {question}\nAssistant: {response}')339 340# single-image single-round conversation (单图单轮对话)341question = '<image>\nPlease describe the image shortly.'342response = model.chat(tokenizer, pixel_values, question, generation_config)343print(f'User: {question}\nAssistant: {response}')344 345# single-image multi-round conversation (单图多轮对话)346question = '<image>\nPlease describe the image in detail.'347response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=None, return_history=True)348print(f'User: {question}\nAssistant: {response}')349 350question = 'Please write a poem according to the image.'351response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=history, return_history=True)352print(f'User: {question}\nAssistant: {response}')353 354# multi-image multi-round conversation, combined images (多图多轮对话，拼接图像)355pixel_values1 = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()356pixel_values2 = load_image('./examples/image2.jpg', max_num=12).to(torch.bfloat16).cuda()357pixel_values = torch.cat((pixel_values1, pixel_values2), dim=0)358 359question = '<image>\nDescribe the two images in detail.'360response, history = model.chat(tokenizer, pixel_values, question, generation_config,361 history=None, return_history=True)362print(f'User: {question}\nAssistant: {response}')363 364question = 'What are the similarities and differences between these two images.'365response, history = model.chat(tokenizer, pixel_values, question, generation_config,366 history=history, return_history=True)367print(f'User: {question}\nAssistant: {response}')368 369# multi-image multi-round conversation, separate images (多图多轮对话，独立图像)370pixel_values1 = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()371pixel_values2 = load_image('./examples/image2.jpg', max_num=12).to(torch.bfloat16).cuda()372pixel_values = torch.cat((pixel_values1, pixel_values2), dim=0)373num_patches_list = [pixel_values1.size(0), pixel_values2.size(0)]374 375question = 'Image-1: <image>\nImage-2: <image>\nDescribe the two images in detail.'376response, history = model.chat(tokenizer, pixel_values, question, generation_config,377 num_patches_list=num_patches_list,378 history=None, return_history=True)379print(f'User: {question}\nAssistant: {response}')380 381question = 'What are the similarities and differences between these two images.'382response, history = model.chat(tokenizer, pixel_values, question, generation_config,383 num_patches_list=num_patches_list,384 history=history, return_history=True)385print(f'User: {question}\nAssistant: {response}')386 387# batch inference, single image per sample (单图批处理)388pixel_values1 = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()389pixel_values2 = load_image('./examples/image2.jpg', max_num=12).to(torch.bfloat16).cuda()390num_patches_list = [pixel_values1.size(0), pixel_values2.size(0)]391pixel_values = torch.cat((pixel_values1, pixel_values2), dim=0)392 393questions = ['<image>\nDescribe the image in detail.'] * len(num_patches_list)394responses = model.batch_chat(tokenizer, pixel_values,395 num_patches_list=num_patches_list,396 questions=questions,397 generation_config=generation_config)398for question, response in zip(questions, responses):399 print(f'User: {question}\nAssistant: {response}')400 401# video multi-round conversation (视频多轮对话)402def get_index(bound, fps, max_frame, first_idx=0, num_segments=32):403 if bound:404 start, end = bound[0], bound[1]405 else:406 start, end = -100000, 100000407 start_idx = max(first_idx, round(start * fps))408 end_idx = min(round(end * fps), max_frame)409 seg_size = float(end_idx - start_idx) / num_segments410 frame_indices = np.array([411 int(start_idx + (seg_size / 2) + np.round(seg_size * idx))412 for idx in range(num_segments)413 ])414 return frame_indices415 416def load_video(video_path, bound=None, input_size=448, max_num=1, num_segments=32):417 vr = VideoReader(video_path, ctx=cpu(0), num_threads=1)418 max_frame = len(vr) - 1419 fps = float(vr.get_avg_fps())420 421 pixel_values_list, num_patches_list = [], []422 transform = build_transform(input_size=input_size)423 frame_indices = get_index(bound, fps, max_frame, first_idx=0, num_segments=num_segments)424 for frame_index in frame_indices:425 img = Image.fromarray(vr[frame_index].asnumpy()).convert('RGB')426 img = dynamic_preprocess(img, image_size=input_size, use_thumbnail=True, max_num=max_num)427 pixel_values = [transform(tile) for tile in img]428 pixel_values = torch.stack(pixel_values)429 num_patches_list.append(pixel_values.shape[0])430 pixel_values_list.append(pixel_values)431 pixel_values = torch.cat(pixel_values_list)432 return pixel_values, num_patches_list433 434video_path = './examples/red-panda.mp4'435pixel_values, num_patches_list = load_video(video_path, num_segments=8, max_num=1)436pixel_values = pixel_values.to(torch.bfloat16).cuda()437video_prefix = ''.join([f'Frame{i+1}: <image>\n' for i in range(len(num_patches_list))])438question = video_prefix + 'What is the red panda doing?'439# Frame1: <image>\nFrame2: <image>\n...\nFrame8: <image>\n{question}440response, history = model.chat(tokenizer, pixel_values, question, generation_config,441 num_patches_list=num_patches_list, history=None, return_history=True)442print(f'User: {question}\nAssistant: {response}')443 444question = 'Describe this video in detail.'445response, history = model.chat(tokenizer, pixel_values, question, generation_config,446 num_patches_list=num_patches_list, history=history, return_history=True)447print(f'User: {question}\nAssistant: {response}')448```449 450#### Streaming Output451 452Besides this method, you can also use the following code to get streamed output.453 454```python455from transformers import TextIteratorStreamer456from threading import Thread457 458# Initialize the streamer459streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True, timeout=10)460# Define the generation configuration461generation_config = dict(max_new_tokens=1024, do_sample=False, streamer=streamer)462# Start the model chat in a separate thread463thread = Thread(target=model.chat, kwargs=dict(464 tokenizer=tokenizer, pixel_values=pixel_values, question=question,465 history=None, return_history=False, generation_config=generation_config,466))467thread.start()468 469# Initialize an empty string to store the generated text470generated_text = ''471# Loop through the streamer to get the new text as it is generated472for new_text in streamer:473 if new_text == model.conv_template.sep:474 break475 generated_text += new_text476 print(new_text, end='', flush=True) # Print each new chunk of generated text on the same line477```478 479## Finetune480 481Many repositories now support fine-tuning of the InternVL series models, including [InternVL](https://github.com/OpenGVLab/InternVL), [SWIFT](https://github.com/modelscope/ms-swift), [XTurner](https://github.com/InternLM/xtuner), and others. Please refer to their documentation for more details on fine-tuning.482 483## Deployment484 485### LMDeploy486 487LMDeploy is a toolkit for compressing, deploying, and serving LLMs & VLMs.488 489```sh490pip install lmdeploy>=0.5.3491```492 493LMDeploy abstracts the complex inference process of multi-modal Vision-Language Models (VLM) into an easy-to-use pipeline, similar to the Large Language Model (LLM) inference pipeline.494 495#### A 'Hello, world' Example496 497```python498from lmdeploy import pipeline, TurbomindEngineConfig499from lmdeploy.vl import load_image500 501model = 'OpenGVLab/InternVL2-40B'502image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')503pipe = pipeline(model, backend_config=TurbomindEngineConfig(session_len=8192, tp=2))504response = pipe(('describe this image', image))505print(response.text)506```507 508If `ImportError` occurs while executing this case, please install the required dependency packages as prompted.509 510#### Multi-images Inference511 512When dealing with multiple images, you can put them all in one list. Keep in mind that multiple images will lead to a higher number of input tokens, and as a result, the size of the context window typically needs to be increased.513 514> Warning: Due to the scarcity of multi-image conversation data, the performance on multi-image tasks may be unstable, and it may require multiple attempts to achieve satisfactory results.515 516```python517from lmdeploy import pipeline, TurbomindEngineConfig518from lmdeploy.vl import load_image519from lmdeploy.vl.constants import IMAGE_TOKEN520 521model = 'OpenGVLab/InternVL2-40B'522pipe = pipeline(model, backend_config=TurbomindEngineConfig(session_len=8192, tp=2))523 524image_urls=[525 'https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/human-pose.jpg',526 'https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/det.jpg'527]528 529images = [load_image(img_url) for img_url in image_urls]530# Numbering images improves multi-image conversations531response = pipe((f'Image-1: {IMAGE_TOKEN}\nImage-2: {IMAGE_TOKEN}\ndescribe these two images', images))532print(response.text)533```534 535#### Batch Prompts Inference536 537Conducting inference with batch prompts is quite straightforward; just place them within a list structure:538 539```python540from lmdeploy import pipeline, TurbomindEngineConfig541from lmdeploy.vl import load_image542 543model = 'OpenGVLab/InternVL2-40B'544pipe = pipeline(model, backend_config=TurbomindEngineConfig(session_len=8192, tp=2))545 546image_urls=[547 "https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/human-pose.jpg",548 "https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/det.jpg"549]550prompts = [('describe this image', load_image(img_url)) for img_url in image_urls]551response = pipe(prompts)552print(response)553```554 555#### Multi-turn Conversation556 557There are two ways to do the multi-turn conversations with the pipeline. One is to construct messages according to the format of OpenAI and use above introduced method, the other is to use the `pipeline.chat` interface.558 559```python560from lmdeploy import pipeline, TurbomindEngineConfig, GenerationConfig561from lmdeploy.vl import load_image562 563model = 'OpenGVLab/InternVL2-40B'564pipe = pipeline(model, backend_config=TurbomindEngineConfig(session_len=8192, tp=2))565 566image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/human-pose.jpg')567gen_config = GenerationConfig(top_k=40, top_p=0.8, temperature=0.8)568sess = pipe.chat(('describe this image', image), gen_config=gen_config)569print(sess.response.text)570sess = pipe.chat('What is the woman doing?', session=sess, gen_config=gen_config)571print(sess.response.text)572```573 574#### Service575 576LMDeploy's `api_server` enables models to be easily packed into services with a single command. The provided RESTful APIs are compatible with OpenAI's interfaces. Below are an example of service startup:577 578```shell579lmdeploy serve api_server OpenGVLab/InternVL2-40B --server-port 23333 --tp 2580```581 582To use the OpenAI-style interface, you need to install OpenAI:583 584```shell585pip install openai586```587 588Then, use the code below to make the API call:589 590```python591from openai import OpenAI592 593client = OpenAI(api_key='YOUR_API_KEY', base_url='http://0.0.0.0:23333/v1')594model_name = client.models.list().data[0].id595response = client.chat.completions.create(596 model=model_name,597 messages=[{598 'role':599 'user',600 'content': [{601 'type': 'text',602 'text': 'describe this image',603 }, {604 'type': 'image_url',605 'image_url': {606 'url':607 'https://modelscope.oss-cn-beijing.aliyuncs.com/resource/tiger.jpeg',608 },609 }],610 }],611 temperature=0.8,612 top_p=0.8)613print(response)614```615 616## License617 618This project is released under the MIT License. This project uses the pre-trained Nous-Hermes-2-Yi-34B as a component, which is licensed under the Apache License 2.0.619 620## Citation621 622If you find this project useful in your research, please consider citing:623 624```BibTeX625@article{chen2024expanding,626 title={Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling},627 author={Chen, Zhe and Wang, Weiyun and Cao, Yue and Liu, Yangzhou and Gao, Zhangwei and Cui, Erfei and Zhu, Jinguo and Ye, Shenglong and Tian, Hao and Liu, Zhaoyang and others},628 journal={arXiv preprint arXiv:2412.05271},629 year={2024}630}631@article{gao2024mini,632 title={Mini-internvl: A flexible-transfer pocket multimodal model with 5\% parameters and 90\% performance},633 author={Gao, Zhangwei and Chen, Zhe and Cui, Erfei and Ren, Yiming and Wang, Weiyun and Zhu, Jinguo and Tian, Hao and Ye, Shenglong and He, Junjun and Zhu, Xizhou and others},634 journal={arXiv preprint arXiv:2410.16261},635 year={2024}636}637@article{chen2024far,638 title={How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites},639 author={Chen, Zhe and Wang, Weiyun and Tian, Hao and Ye, Shenglong and Gao, Zhangwei and Cui, Erfei and Tong, Wenwen and Hu, Kongzhi and Luo, Jiapeng and Ma, Zheng and others},640 journal={arXiv preprint arXiv:2404.16821},641 year={2024}642}643@inproceedings{chen2024internvl,644 title={Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks},645 author={Chen, Zhe and Wu, Jiannan and Wang, Wenhai and Su, Weijie and Chen, Guo and Xing, Sen and Zhong, Muyan and Zhang, Qinglong and Zhu, Xizhou and Lu, Lewei and others},646 booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},647 pages={24185--24198},648 year={2024}649}650```651