FriendliAI/InternVL2_5-2B
019
1---2license: mit3pipeline_tag: image-text-to-text4library_name: transformers5base_model:6 - OpenGVLab/InternViT-300M-448px-V2_57 - internlm/internlm2_5-1_8b-chat8base_model_relation: merge9language:10 - multilingual11tags:12 - internvl13 - custom_code14datasets:15 - HuggingFaceFV/finevideo16---17 18# InternVL2_5-2B19 20[\[📂 GitHub\]](https://github.com/OpenGVLab/InternVL) [\[📜 InternVL 1.0\]](https://huggingface.co/papers/2312.14238) [\[📜 InternVL 1.5\]](https://huggingface.co/papers/2404.16821) [\[📜 Mini-InternVL\]](https://arxiv.org/abs/2410.16261) [\[📜 InternVL 2.5\]](https://huggingface.co/papers/2412.05271)21 22[\[🆕 Blog\]](https://internvl.github.io/blog/) [\[🗨️ Chat Demo\]](https://internvl.opengvlab.com/) [\[🤗 HF Demo\]](https://huggingface.co/spaces/OpenGVLab/InternVL) [\[🚀 Quick Start\]](#quick-start) [\[📖 Documents\]](https://internvl.readthedocs.io/en/latest/)23 24<div align="center">25 <img width="500" alt="image" src="https://cdn-uploads.huggingface.co/production/uploads/64006c09330a45b03605bba3/zJsd2hqd3EevgXo6fNgC-.png">26</div>27 28## Introduction29 30We are excited to introduce **InternVL 2.5**, an advanced multimodal large language model (MLLM) series that builds upon InternVL 2.0, maintaining its core model architecture while introducing significant enhancements in training and testing strategies as well as data quality.31 3233 34## InternVL 2.5 Family35 36In the following table, we provide an overview of the InternVL 2.5 series.37 38| Model Name | Vision Part | Language Part | HF Link |39| :-------------: | :-------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------: | :---------------------------------------------------------: |40| InternVL2_5-1B | [InternViT-300M-448px-V2_5](https://huggingface.co/OpenGVLab/InternViT-300M-448px-V2_5) | [Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2_5-1B) |41| InternVL2_5-2B | [InternViT-300M-448px-V2_5](https://huggingface.co/OpenGVLab/InternViT-300M-448px-V2_5) | [internlm2_5-1_8b-chat](https://huggingface.co/internlm/internlm2_5-1_8b-chat) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2_5-2B) |42| InternVL2_5-4B | [InternViT-300M-448px-V2_5](https://huggingface.co/OpenGVLab/InternViT-300M-448px-V2_5) | [Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2_5-4B) |43| InternVL2_5-8B | [InternViT-300M-448px-V2_5](https://huggingface.co/OpenGVLab/InternViT-300M-448px-V2_5) | [internlm2_5-7b-chat](https://huggingface.co/internlm/internlm2_5-7b-chat) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2_5-8B) |44| InternVL2_5-26B | [InternViT-6B-448px-V2_5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V2_5) | [internlm2_5-20b-chat](https://huggingface.co/internlm/internlm2_5-20b-chat) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2_5-26B) |45| InternVL2_5-38B | [InternViT-6B-448px-V2_5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V2_5) | [Qwen2.5-32B-Instruct](https://huggingface.co/Qwen/Qwen2.5-32B-Instruct) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2_5-38B) |46| InternVL2_5-78B | [InternViT-6B-448px-V2_5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V2_5) | [Qwen2.5-72B-Instruct](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct) | [🤗 link](https://huggingface.co/OpenGVLab/InternVL2_5-78B) |47 48## Model Architecture49 50As shown in the following figure, InternVL 2.5 retains the same model architecture as its predecessors, InternVL 1.5 and 2.0, following the "ViT-MLP-LLM" paradigm. In this new version, we integrate a newly incrementally pre-trained InternViT with various pre-trained LLMs, including InternLM 2.5 and Qwen 2.5, using a randomly initialized MLP projector.51 5253 54As in the previous version, we applied a pixel unshuffle operation, reducing the number of visual tokens to one-quarter of the original. Besides, we adopted a similar dynamic resolution strategy as InternVL 1.5, dividing images into tiles of 448×448 pixels. The key difference, starting from InternVL 2.0, is that we additionally introduced support for multi-image and video data.55 56## Training Strategy57 58### Dynamic High-Resolution for Multimodal Data59 60In InternVL 2.0 and 2.5, we extend the dynamic high-resolution training approach, enhancing its capabilities to handle multi-image and video datasets.61 6263 64- For single-image datasets, the total number of tiles `n_max` are allocated to a single image for maximum resolution. Visual tokens are enclosed in `<img>` and `</img>` tags.65 66- For multi-image datasets, the total number of tiles `n_max` are distributed across all images in a sample. Each image is labeled with auxiliary tags like `Image-1` and enclosed in `<img>` and `</img>` tags.67 68- For videos, each frame is resized to 448×448. Frames are labeled with tags like `Frame-1` and enclosed in `<img>` and `</img>` tags, similar to images.69 70### Single Model Training Pipeline71 72The training pipeline for a single model in InternVL 2.5 is structured across three stages, designed to enhance the model's visual perception and multimodal capabilities.73 7475 76- **Stage 1: MLP Warmup.** In this stage, only the MLP projector is trained while the vision encoder and language model are frozen. A dynamic high-resolution training strategy is applied for better performance, despite increased cost. This phase ensures robust cross-modal alignment and prepares the model for stable multimodal training.77 78- **Stage 1.5: ViT Incremental Learning (Optional).** This stage allows incremental training of the vision encoder and MLP projector using the same data as Stage 1. It enhances the encoder’s ability to handle rare domains like multilingual OCR and mathematical charts. Once trained, the encoder can be reused across LLMs without retraining, making this stage optional unless new domains are introduced.79 80- **Stage 2: Full Model Instruction Tuning.** The entire model is trained on high-quality multimodal instruction datasets. Strict data quality controls are enforced to prevent degradation of the LLM, as noisy data can cause issues like repetitive or incorrect outputs. After this stage, the training process is complete.81 82### Progressive Scaling Strategy83 84We introduce a progressive scaling strategy to align the vision encoder with LLMs efficiently. This approach trains with smaller LLMs first (e.g., 20B) to optimize foundational visual capabilities and cross-modal alignment before transferring the vision encoder to larger LLMs (e.g., 72B) without retraining. This reuse skips intermediate stages for larger models.85 8687 88Compared to Qwen2-VL's 1.4 trillion tokens, InternVL2.5-78B uses only 120 billion tokens—less than one-tenth. This strategy minimizes redundancy, maximizes pre-trained component reuse, and enables efficient training for complex vision-language tasks.89 90### Training Enhancements91 92To improve real-world adaptability and performance, we introduce two key techniques:93 94- **Random JPEG Compression**: Random JPEG compression with quality levels between 75 and 100 is applied as a data augmentation technique. This simulates image degradation from internet sources, enhancing the model's robustness to noisy images.95 96- **Loss Reweighting**: To balance the NTP loss across responses of different lengths, we use a reweighting strategy called **square averaging**. This method balances contributions from responses of varying lengths, mitigating biases toward longer or shorter responses.97 98### Data Organization99 100#### Dataset Configuration101 102In InternVL 2.0 and 2.5, the organization of the training data is controlled by several key parameters to optimize the balance and distribution of datasets during training.103 104105 106- **Data Augmentation:** JPEG compression is applied conditionally: enabled for image datasets to enhance robustness and disabled for video datasets to maintain consistent frame quality.107 108- **Maximum Tile Number:** The parameter `n_max` controls the maximum tiles per dataset. For example, higher values (24–36) are used for multi-image or high-resolution data, lower values (6–12) for standard images, and 1 for videos.109 110- **Repeat Factor:** The repeat factor `r` adjusts dataset sampling frequency. Values below 1 reduce a dataset's weight, while values above 1 increase it. This ensures balanced training across tasks and prevents overfitting or underfitting.111 112#### Data Filtering Pipeline113 114During development, we found that LLMs are highly sensitive to data noise, with even small anomalies—like outliers or repetitive data—causing abnormal behavior during inference. Repetitive generation, especially in long-form or CoT reasoning tasks, proved particularly harmful.115 116117 118To address this challenge and support future research, we designed an efficient data filtering pipeline to remove low-quality samples.119 120121 122The pipeline includes two modules, for **pure-text data**, three key strategies are used:123 1241. **LLM-Based Quality Scoring**: Each sample is scored (0–10) using a pre-trained LLM with domain-specific prompts. Samples scoring below a threshold (e.g., 7) are removed to ensure high-quality data.1252. **Repetition Detection**: Repetitive samples are flagged using LLM-based prompts and manually reviewed. Samples scoring below a stricter threshold (e.g., 3) are excluded to avoid repetitive patterns.1263. **Heuristic Rule-Based Filtering**: Anomalies like abnormal sentence lengths or duplicate lines are detected using rules. Flagged samples undergo manual verification to ensure accuracy before removal.127 128For **multimodal data**, two strategies are used:129 1301. **Repetition Detection**: Repetitive samples in non-academic datasets are flagged and manually reviewed to prevent pattern loops. High-quality datasets are exempt from this process.1312. **Heuristic Rule-Based Filtering**: Similar rules are applied to detect visual anomalies, with flagged data verified manually to maintain integrity.132 133#### Training Data134 135As shown in the following figure, from InternVL 1.5 to 2.0 and then to 2.5, the fine-tuning data mixture has undergone iterative improvements in scale, quality, and diversity. For more information about the training data, please refer to our technical report.136 137138 139## Evaluation on Multimodal Capability140 141### Multimodal Reasoning and Mathematics142 143144 145146 147### OCR, Chart, and Document Understanding148 149150 151### Multi-Image & Real-World Comprehension152 153154 155### Comprehensive Multimodal & Hallucination Evaluation156 157158 159### Visual Grounding160 161162 163### Multimodal Multilingual Understanding164 165166 167### Video Understanding168 169170 171## Evaluation on Language Capability172 173Training InternVL 2.0 models led to a decline in pure language capabilities. InternVL 2.5 addresses this by collecting more high-quality open-source data and filtering out low-quality data, achieving better preservation of pure language performance.174 175176 177## Quick Start178 179We provide an example code to run `InternVL2_5-2B` using `transformers`.180 181> Please use transformers>=4.37.2 to ensure the model works normally.182 183### Model Loading184 185#### 16-bit (bf16 / fp16)186 187```python188import torch189from transformers import AutoTokenizer, AutoModel190path = "OpenGVLab/InternVL2_5-2B"191model = AutoModel.from_pretrained(192 path,193 torch_dtype=torch.bfloat16,194 low_cpu_mem_usage=True,195 use_flash_attn=True,196 trust_remote_code=True).eval().cuda()197```198 199#### BNB 8-bit Quantization200 201```python202import torch203from transformers import AutoTokenizer, AutoModel204path = "OpenGVLab/InternVL2_5-2B"205model = AutoModel.from_pretrained(206 path,207 torch_dtype=torch.bfloat16,208 load_in_8bit=True,209 low_cpu_mem_usage=True,210 use_flash_attn=True,211 trust_remote_code=True).eval()212```213 214#### Multiple GPUs215 216The reason for writing the code this way is to avoid errors that occur during multi-GPU inference due to tensors not being on the same device. By ensuring that the first and last layers of the large language model (LLM) are on the same device, we prevent such errors.217 218```python219import math220import torch221from transformers import AutoTokenizer, AutoModel222 223def split_model(model_name):224 device_map = {}225 world_size = torch.cuda.device_count()226 num_layers = {227 'InternVL2_5-1B': 24, 'InternVL2_5-2B': 24, 'InternVL2_5-4B': 36, 'InternVL2_5-8B': 32,228 'InternVL2_5-26B': 48, 'InternVL2_5-38B': 64, 'InternVL2_5-78B': 80}[model_name]229 # Since the first GPU will be used for ViT, treat it as half a GPU.230 num_layers_per_gpu = math.ceil(num_layers / (world_size - 0.5))231 num_layers_per_gpu = [num_layers_per_gpu] * world_size232 num_layers_per_gpu[0] = math.ceil(num_layers_per_gpu[0] * 0.5)233 layer_cnt = 0234 for i, num_layer in enumerate(num_layers_per_gpu):235 for j in range(num_layer):236 device_map[f'language_model.model.layers.{layer_cnt}'] = i237 layer_cnt += 1238 device_map['vision_model'] = 0239 device_map['mlp1'] = 0240 device_map['language_model.model.tok_embeddings'] = 0241 device_map['language_model.model.embed_tokens'] = 0242 device_map['language_model.output'] = 0243 device_map['language_model.model.norm'] = 0244 device_map['language_model.model.rotary_emb'] = 0245 device_map['language_model.lm_head'] = 0246 device_map[f'language_model.model.layers.{num_layers - 1}'] = 0247 248 return device_map249 250path = "OpenGVLab/InternVL2_5-2B"251device_map = split_model('InternVL2_5-2B')252model = AutoModel.from_pretrained(253 path,254 torch_dtype=torch.bfloat16,255 low_cpu_mem_usage=True,256 use_flash_attn=True,257 trust_remote_code=True,258 device_map=device_map).eval()259```260 261### Inference with Transformers262 263```python264import numpy as np265import torch266import torchvision.transforms as T267from decord import VideoReader, cpu268from PIL import Image269from torchvision.transforms.functional import InterpolationMode270from transformers import AutoModel, AutoTokenizer271 272IMAGENET_MEAN = (0.485, 0.456, 0.406)273IMAGENET_STD = (0.229, 0.224, 0.225)274 275def build_transform(input_size):276 MEAN, STD = IMAGENET_MEAN, IMAGENET_STD277 transform = T.Compose([278 T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img),279 T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),280 T.ToTensor(),281 T.Normalize(mean=MEAN, std=STD)282 ])283 return transform284 285def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):286 best_ratio_diff = float('inf')287 best_ratio = (1, 1)288 area = width * height289 for ratio in target_ratios:290 target_aspect_ratio = ratio[0] / ratio[1]291 ratio_diff = abs(aspect_ratio - target_aspect_ratio)292 if ratio_diff < best_ratio_diff:293 best_ratio_diff = ratio_diff294 best_ratio = ratio295 elif ratio_diff == best_ratio_diff:296 if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]:297 best_ratio = ratio298 return best_ratio299 300def dynamic_preprocess(image, min_num=1, max_num=12, image_size=448, use_thumbnail=False):301 orig_width, orig_height = image.size302 aspect_ratio = orig_width / orig_height303 304 # calculate the existing image aspect ratio305 target_ratios = set(306 (i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if307 i * j <= max_num and i * j >= min_num)308 target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])309 310 # find the closest aspect ratio to the target311 target_aspect_ratio = find_closest_aspect_ratio(312 aspect_ratio, target_ratios, orig_width, orig_height, image_size)313 314 # calculate the target width and height315 target_width = image_size * target_aspect_ratio[0]316 target_height = image_size * target_aspect_ratio[1]317 blocks = target_aspect_ratio[0] * target_aspect_ratio[1]318 319 # resize the image320 resized_img = image.resize((target_width, target_height))321 processed_images = []322 for i in range(blocks):323 box = (324 (i % (target_width // image_size)) * image_size,325 (i // (target_width // image_size)) * image_size,326 ((i % (target_width // image_size)) + 1) * image_size,327 ((i // (target_width // image_size)) + 1) * image_size328 )329 # split the image330 split_img = resized_img.crop(box)331 processed_images.append(split_img)332 assert len(processed_images) == blocks333 if use_thumbnail and len(processed_images) != 1:334 thumbnail_img = image.resize((image_size, image_size))335 processed_images.append(thumbnail_img)336 return processed_images337 338def load_image(image_file, input_size=448, max_num=12):339 image = Image.open(image_file).convert('RGB')340 transform = build_transform(input_size=input_size)341 images = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num)342 pixel_values = [transform(image) for image in images]343 pixel_values = torch.stack(pixel_values)344 return pixel_values345 346# If you want to load a model using multiple GPUs, please refer to the `Multiple GPUs` section.347path = 'OpenGVLab/InternVL2_5-2B'348model = AutoModel.from_pretrained(349 path,350 torch_dtype=torch.bfloat16,351 low_cpu_mem_usage=True,352 use_flash_attn=True,353 trust_remote_code=True).eval().cuda()354tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True, use_fast=False)355 356# set the max number of tiles in `max_num`357pixel_values = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()358generation_config = dict(max_new_tokens=1024, do_sample=True)359 360# pure-text conversation (纯文本对话)361question = 'Hello, who are you?'362response, history = model.chat(tokenizer, None, question, generation_config, history=None, return_history=True)363print(f'User: {question}\nAssistant: {response}')364 365question = 'Can you tell me a story?'366response, history = model.chat(tokenizer, None, question, generation_config, history=history, return_history=True)367print(f'User: {question}\nAssistant: {response}')368 369# single-image single-round conversation (单图单轮对话)370question = '<image>\nPlease describe the image shortly.'371response = model.chat(tokenizer, pixel_values, question, generation_config)372print(f'User: {question}\nAssistant: {response}')373 374# single-image multi-round conversation (单图多轮对话)375question = '<image>\nPlease describe the image in detail.'376response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=None, return_history=True)377print(f'User: {question}\nAssistant: {response}')378 379question = 'Please write a poem according to the image.'380response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=history, return_history=True)381print(f'User: {question}\nAssistant: {response}')382 383# multi-image multi-round conversation, combined images (多图多轮对话，拼接图像)384pixel_values1 = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()385pixel_values2 = load_image('./examples/image2.jpg', max_num=12).to(torch.bfloat16).cuda()386pixel_values = torch.cat((pixel_values1, pixel_values2), dim=0)387 388question = '<image>\nDescribe the two images in detail.'389response, history = model.chat(tokenizer, pixel_values, question, generation_config,390 history=None, return_history=True)391print(f'User: {question}\nAssistant: {response}')392 393question = 'What are the similarities and differences between these two images.'394response, history = model.chat(tokenizer, pixel_values, question, generation_config,395 history=history, return_history=True)396print(f'User: {question}\nAssistant: {response}')397 398# multi-image multi-round conversation, separate images (多图多轮对话，独立图像)399pixel_values1 = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()400pixel_values2 = load_image('./examples/image2.jpg', max_num=12).to(torch.bfloat16).cuda()401pixel_values = torch.cat((pixel_values1, pixel_values2), dim=0)402num_patches_list = [pixel_values1.size(0), pixel_values2.size(0)]403 404question = 'Image-1: <image>\nImage-2: <image>\nDescribe the two images in detail.'405response, history = model.chat(tokenizer, pixel_values, question, generation_config,406 num_patches_list=num_patches_list,407 history=None, return_history=True)408print(f'User: {question}\nAssistant: {response}')409 410question = 'What are the similarities and differences between these two images.'411response, history = model.chat(tokenizer, pixel_values, question, generation_config,412 num_patches_list=num_patches_list,413 history=history, return_history=True)414print(f'User: {question}\nAssistant: {response}')415 416# batch inference, single image per sample (单图批处理)417pixel_values1 = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()418pixel_values2 = load_image('./examples/image2.jpg', max_num=12).to(torch.bfloat16).cuda()419num_patches_list = [pixel_values1.size(0), pixel_values2.size(0)]420pixel_values = torch.cat((pixel_values1, pixel_values2), dim=0)421 422questions = ['<image>\nDescribe the image in detail.'] * len(num_patches_list)423responses = model.batch_chat(tokenizer, pixel_values,424 num_patches_list=num_patches_list,425 questions=questions,426 generation_config=generation_config)427for question, response in zip(questions, responses):428 print(f'User: {question}\nAssistant: {response}')429 430# video multi-round conversation (视频多轮对话)431def get_index(bound, fps, max_frame, first_idx=0, num_segments=32):432 if bound:433 start, end = bound[0], bound[1]434 else:435 start, end = -100000, 100000436 start_idx = max(first_idx, round(start * fps))437 end_idx = min(round(end * fps), max_frame)438 seg_size = float(end_idx - start_idx) / num_segments439 frame_indices = np.array([440 int(start_idx + (seg_size / 2) + np.round(seg_size * idx))441 for idx in range(num_segments)442 ])443 return frame_indices444 445def load_video(video_path, bound=None, input_size=448, max_num=1, num_segments=32):446 vr = VideoReader(video_path, ctx=cpu(0), num_threads=1)447 max_frame = len(vr) - 1448 fps = float(vr.get_avg_fps())449 450 pixel_values_list, num_patches_list = [], []451 transform = build_transform(input_size=input_size)452 frame_indices = get_index(bound, fps, max_frame, first_idx=0, num_segments=num_segments)453 for frame_index in frame_indices:454 img = Image.fromarray(vr[frame_index].asnumpy()).convert('RGB')455 img = dynamic_preprocess(img, image_size=input_size, use_thumbnail=True, max_num=max_num)456 pixel_values = [transform(tile) for tile in img]457 pixel_values = torch.stack(pixel_values)458 num_patches_list.append(pixel_values.shape[0])459 pixel_values_list.append(pixel_values)460 pixel_values = torch.cat(pixel_values_list)461 return pixel_values, num_patches_list462 463video_path = './examples/red-panda.mp4'464pixel_values, num_patches_list = load_video(video_path, num_segments=8, max_num=1)465pixel_values = pixel_values.to(torch.bfloat16).cuda()466video_prefix = ''.join([f'Frame{i+1}: <image>\n' for i in range(len(num_patches_list))])467question = video_prefix + 'What is the red panda doing?'468# Frame1: <image>\nFrame2: <image>\n...\nFrame8: <image>\n{question}469response, history = model.chat(tokenizer, pixel_values, question, generation_config,470 num_patches_list=num_patches_list, history=None, return_history=True)471print(f'User: {question}\nAssistant: {response}')472 473question = 'Describe this video in detail.'474response, history = model.chat(tokenizer, pixel_values, question, generation_config,475 num_patches_list=num_patches_list, history=history, return_history=True)476print(f'User: {question}\nAssistant: {response}')477```478 479#### Streaming Output480 481Besides this method, you can also use the following code to get streamed output.482 483```python484from transformers import TextIteratorStreamer485from threading import Thread486 487# Initialize the streamer488streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True, timeout=10)489# Define the generation configuration490generation_config = dict(max_new_tokens=1024, do_sample=False, streamer=streamer)491# Start the model chat in a separate thread492thread = Thread(target=model.chat, kwargs=dict(493 tokenizer=tokenizer, pixel_values=pixel_values, question=question,494 history=None, return_history=False, generation_config=generation_config,495))496thread.start()497 498# Initialize an empty string to store the generated text499generated_text = ''500# Loop through the streamer to get the new text as it is generated501for new_text in streamer:502 if new_text == model.conv_template.sep:503 break504 generated_text += new_text505 print(new_text, end='', flush=True) # Print each new chunk of generated text on the same line506```507 508## Finetune509 510Many repositories now support fine-tuning of the InternVL series models, including [InternVL](https://github.com/OpenGVLab/InternVL), [SWIFT](https://github.com/modelscope/ms-swift), [XTurner](https://github.com/InternLM/xtuner), and others. Please refer to their documentation for more details on fine-tuning.511 512## Deployment513 514### LMDeploy515 516LMDeploy is a toolkit for compressing, deploying, and serving LLMs & VLMs.517 518```sh519pip install lmdeploy>=0.6.4520```521 522LMDeploy abstracts the complex inference process of multi-modal Vision-Language Models (VLM) into an easy-to-use pipeline, similar to the Large Language Model (LLM) inference pipeline.523 524#### A 'Hello, world' Example525 526```python527from lmdeploy import pipeline, TurbomindEngineConfig528from lmdeploy.vl import load_image529 530model = 'OpenGVLab/InternVL2_5-2B'531image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')532pipe = pipeline(model, backend_config=TurbomindEngineConfig(session_len=8192))533response = pipe(('describe this image', image))534print(response.text)535```536 537If `ImportError` occurs while executing this case, please install the required dependency packages as prompted.538 539#### Multi-images Inference540 541When dealing with multiple images, you can put them all in one list. Keep in mind that multiple images will lead to a higher number of input tokens, and as a result, the size of the context window typically needs to be increased.542 543```python544from lmdeploy import pipeline, TurbomindEngineConfig545from lmdeploy.vl import load_image546from lmdeploy.vl.constants import IMAGE_TOKEN547 548model = 'OpenGVLab/InternVL2_5-2B'549pipe = pipeline(model, backend_config=TurbomindEngineConfig(session_len=8192))550 551image_urls=[552 'https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/human-pose.jpg',553 'https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/det.jpg'554]555 556images = [load_image(img_url) for img_url in image_urls]557# Numbering images improves multi-image conversations558response = pipe((f'Image-1: {IMAGE_TOKEN}\nImage-2: {IMAGE_TOKEN}\ndescribe these two images', images))559print(response.text)560```561 562#### Batch Prompts Inference563 564Conducting inference with batch prompts is quite straightforward; just place them within a list structure:565 566```python567from lmdeploy import pipeline, TurbomindEngineConfig568from lmdeploy.vl import load_image569 570model = 'OpenGVLab/InternVL2_5-2B'571pipe = pipeline(model, backend_config=TurbomindEngineConfig(session_len=8192))572 573image_urls=[574 "https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/human-pose.jpg",575 "https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/det.jpg"576]577prompts = [('describe this image', load_image(img_url)) for img_url in image_urls]578response = pipe(prompts)579print(response)580```581 582#### Multi-turn Conversation583 584There are two ways to do the multi-turn conversations with the pipeline. One is to construct messages according to the format of OpenAI and use above introduced method, the other is to use the `pipeline.chat` interface.585 586```python587from lmdeploy import pipeline, TurbomindEngineConfig, GenerationConfig588from lmdeploy.vl import load_image589 590model = 'OpenGVLab/InternVL2_5-2B'591pipe = pipeline(model, backend_config=TurbomindEngineConfig(session_len=8192))592 593image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/demo/resources/human-pose.jpg')594gen_config = GenerationConfig(top_k=40, top_p=0.8, temperature=0.8)595sess = pipe.chat(('describe this image', image), gen_config=gen_config)596print(sess.response.text)597sess = pipe.chat('What is the woman doing?', session=sess, gen_config=gen_config)598print(sess.response.text)599```600 601#### Service602 603LMDeploy's `api_server` enables models to be easily packed into services with a single command. The provided RESTful APIs are compatible with OpenAI's interfaces. Below are an example of service startup:604 605```shell606lmdeploy serve api_server OpenGVLab/InternVL2_5-2B --server-port 23333607```608 609To use the OpenAI-style interface, you need to install OpenAI:610 611```shell612pip install openai613```614 615Then, use the code below to make the API call:616 617```python618from openai import OpenAI619 620client = OpenAI(api_key='YOUR_API_KEY', base_url='http://0.0.0.0:23333/v1')621model_name = client.models.list().data[0].id622response = client.chat.completions.create(623 model=model_name,624 messages=[{625 'role':626 'user',627 'content': [{628 'type': 'text',629 'text': 'describe this image',630 }, {631 'type': 'image_url',632 'image_url': {633 'url':634 'https://modelscope.oss-cn-beijing.aliyuncs.com/resource/tiger.jpeg',635 },636 }],637 }],638 temperature=0.8,639 top_p=0.8)640print(response)641```642 643## License644 645This project is released under the MIT License. This project uses the pre-trained internlm2_5-1_8b-chat as a component, which is licensed under the Apache License 2.0.646 647## Citation648 649If you find this project useful in your research, please consider citing:650 651```BibTeX652@article{chen2024expanding,653 title={Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling},654 author={Chen, Zhe and Wang, Weiyun and Cao, Yue and Liu, Yangzhou and Gao, Zhangwei and Cui, Erfei and Zhu, Jinguo and Ye, Shenglong and Tian, Hao and Liu, Zhaoyang and others},655 journal={arXiv preprint arXiv:2412.05271},656 year={2024}657}658@article{gao2024mini,659 title={Mini-internvl: A flexible-transfer pocket multimodal model with 5\% parameters and 90\% performance},660 author={Gao, Zhangwei and Chen, Zhe and Cui, Erfei and Ren, Yiming and Wang, Weiyun and Zhu, Jinguo and Tian, Hao and Ye, Shenglong and He, Junjun and Zhu, Xizhou and others},661 journal={arXiv preprint arXiv:2410.16261},662 year={2024}663}664@article{chen2024far,665 title={How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites},666 author={Chen, Zhe and Wang, Weiyun and Tian, Hao and Ye, Shenglong and Gao, Zhangwei and Cui, Erfei and Tong, Wenwen and Hu, Kongzhi and Luo, Jiapeng and Ma, Zheng and others},667 journal={arXiv preprint arXiv:2404.16821},668 year={2024}669}670@inproceedings{chen2024internvl,671 title={Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks},672 author={Chen, Zhe and Wu, Jiannan and Wang, Wenhai and Su, Weijie and Chen, Guo and Xing, Sen and Zhong, Muyan and Zhang, Qinglong and Zhu, Xizhou and Lu, Lewei and others},673 booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},674 pages={24185--24198},675 year={2024}676}677```678