OpenGVLab/InternVideo2_5_Chat_8B

1---2language:3- en4library_name: transformers5license: apache-2.06metrics:7- accuracy8tags:9- multimodal10pipeline_tag: video-text-to-text11model-index:12- name: InternVideo2.513  results:14  - task:15      type: multimodal16    dataset:17      name: MLVU18      type: mlvu19    metrics:20    - type: accuracy21      value: 72.822      name: accuracy23      verified: true24  - task:25      type: multimodal26    dataset:27      name: MVBench28      type: mvbench29    metrics:30    - type: accuracy31      value: 75.732      name: accuracy33      verified: true34  - task:35      type: multimodal36    dataset:37      name: Perception Test38      type: percepTest39    metrics:40    - type: accuracy41      value: 74.942      name: accuracy43      verified: true44  - task:45      type: multimodal46    dataset:47      name: LongVideoBench48      type: longvideobench49    metrics:50    - type: accuracy51      value: 60.652      name: accuracy53      verified: true54  - task:55      type: multimodal56    dataset:57      name: VideoMME (w/o sub)58      type: videomme59    metrics:60    - type: accuracy61      value: 65.162      name: accuracy63      verified: true64  - task:65      type: multimodal66    dataset:67      name: LVBench68      type: lvbench69    metrics:70    - type: accuracy71      value: 46.472      name: accuracy73      verified: true74 75 76---77 78# 📕InternVideo2.5⚡79<!-- [\[📰 Blog\]](https://internvideo.github.io/blog/2024-12-31-VideoChat-Flash) -->80[\[📂 GitHub\]](https://github.com/OpenGVLab/InternVideo/tree/main/InternVideo2.5)  81[\[📜 Tech Report\]](https://arxiv.org/abs/2501.12386) 82<!-- [\[🗨️ Chat Demo\]](https://huggingface.co/spaces/OpenGVLab/VideoChat-Flash) -->83 84 InternVideo2.5 is a video multimodal large language model (MLLM, built upoon InternVL2.5) enhanced with **long and rich context (LRC) modeling**. It significantly improves upon existing MLLMs by enhancing their ability to perceive fine-grained details and capture long-form temporal structures. We achieve this through dense vision task annotations using direct preference optimization (TPO) and compact spatiotemporal representations via adaptive hierarchical token compression (HiCo).85 86 87 88 89## 📈 Performance90 91- VideoBenchmark92| Model |  MVBench | LongVideoBench |  VideoMME(w/o sub)| 93| ---   |  ---     |   ---            | ---     | 94|InternVideo2.5| 75.7 |  60.6   | 65.1| 95 96- Inference Speed97 98We measured the average inference speed (tokens/s) of generating 1024 new tokens and 5198 (8192-2998) tokens with the context of an video (which takes 2998 tokens) under BF16 precision. w/ encoder indicates that the inference includes the time for video encoder.99 100|Quantization |	Speed (3022 tokens)	| Speed (8192 tokens) w/o encoder| Speed(8192 tokens) w/ encoder|101|--- |--- |---| ---|102|BF16 |	33.40 |	31.91 | 21.33|103|INT4 | -     | 31.95 | 26.37|104 105The profiling runs on a single A800-SXM4-80G GPU with PyTorch 2.4.0 and CUDA 12.1.106 107 108## 🚀 How to use the model109 110First, you need to install [flash attention2](https://github.com/Dao-AILab/flash-attention) and some other modules. We provide a simple installation example below:111```112pip install transformers==4.40.1113pip install av114pip install imageio115pip install decord116pip install opencv-python117pip install flash-attn --no-build-isolation118```119Then you could use our model:120```python121import numpy as np122import torch123import torchvision.transforms as T124from decord import VideoReader, cpu125from PIL import Image126from torchvision.transforms.functional import InterpolationMode127from transformers import AutoModel, AutoTokenizer128 129 130# model setting131model_path = 'OpenGVLab/InternVideo2_5_Chat_8B'132 133tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)134model = AutoModel.from_pretrained(model_path, trust_remote_code=True).half().cuda().to(torch.bfloat16)135 136IMAGENET_MEAN = (0.485, 0.456, 0.406)137IMAGENET_STD = (0.229, 0.224, 0.225)138 139def build_transform(input_size):140    MEAN, STD = IMAGENET_MEAN, IMAGENET_STD141    transform = T.Compose([T.Lambda(lambda img: img.convert("RGB") if img.mode != "RGB" else img), T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC), T.ToTensor(), T.Normalize(mean=MEAN, std=STD)])142    return transform143 144 145def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):146    best_ratio_diff = float("inf")147    best_ratio = (1, 1)148    area = width * height149    for ratio in target_ratios:150        target_aspect_ratio = ratio[0] / ratio[1]151        ratio_diff = abs(aspect_ratio - target_aspect_ratio)152        if ratio_diff < best_ratio_diff:153            best_ratio_diff = ratio_diff154            best_ratio = ratio155        elif ratio_diff == best_ratio_diff:156            if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]:157                best_ratio = ratio158    return best_ratio159 160 161def dynamic_preprocess(image, min_num=1, max_num=6, image_size=448, use_thumbnail=False):162    orig_width, orig_height = image.size163    aspect_ratio = orig_width / orig_height164 165    # calculate the existing image aspect ratio166    target_ratios = set((i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if i * j <= max_num and i * j >= min_num)167    target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])168 169    # find the closest aspect ratio to the target170    target_aspect_ratio = find_closest_aspect_ratio(aspect_ratio, target_ratios, orig_width, orig_height, image_size)171 172    # calculate the target width and height173    target_width = image_size * target_aspect_ratio[0]174    target_height = image_size * target_aspect_ratio[1]175    blocks = target_aspect_ratio[0] * target_aspect_ratio[1]176 177    # resize the image178    resized_img = image.resize((target_width, target_height))179    processed_images = []180    for i in range(blocks):181        box = ((i % (target_width // image_size)) * image_size, (i // (target_width // image_size)) * image_size, ((i % (target_width // image_size)) + 1) * image_size, ((i // (target_width // image_size)) + 1) * image_size)182        # split the image183        split_img = resized_img.crop(box)184        processed_images.append(split_img)185    assert len(processed_images) == blocks186    if use_thumbnail and len(processed_images) != 1:187        thumbnail_img = image.resize((image_size, image_size))188        processed_images.append(thumbnail_img)189    return processed_images190 191 192def load_image(image, input_size=448, max_num=6):193    transform = build_transform(input_size=input_size)194    images = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num)195    pixel_values = [transform(image) for image in images]196    pixel_values = torch.stack(pixel_values)197    return pixel_values198 199 200def get_index(bound, fps, max_frame, first_idx=0, num_segments=32):201    if bound:202        start, end = bound[0], bound[1]203    else:204        start, end = -100000, 100000205    start_idx = max(first_idx, round(start * fps))206    end_idx = min(round(end * fps), max_frame)207    seg_size = float(end_idx - start_idx) / num_segments208    frame_indices = np.array([int(start_idx + (seg_size / 2) + np.round(seg_size * idx)) for idx in range(num_segments)])209    return frame_indices210 211def get_num_frames_by_duration(duration):212        local_num_frames = 4        213        num_segments = int(duration // local_num_frames)214        if num_segments == 0:215            num_frames = local_num_frames216        else:217            num_frames = local_num_frames * num_segments218        219        num_frames = min(512, num_frames)220        num_frames = max(128, num_frames)221 222        return num_frames223 224def load_video(video_path, bound=None, input_size=448, max_num=1, num_segments=32, get_frame_by_duration = False):225    vr = VideoReader(video_path, ctx=cpu(0), num_threads=1)226    max_frame = len(vr) - 1227    fps = float(vr.get_avg_fps())228 229    pixel_values_list, num_patches_list = [], []230    transform = build_transform(input_size=input_size)231    if get_frame_by_duration:232        duration = max_frame / fps233        num_segments = get_num_frames_by_duration(duration)234    frame_indices = get_index(bound, fps, max_frame, first_idx=0, num_segments=num_segments)235    for frame_index in frame_indices:236        img = Image.fromarray(vr[frame_index].asnumpy()).convert("RGB")237        img = dynamic_preprocess(img, image_size=input_size, use_thumbnail=True, max_num=max_num)238        pixel_values = [transform(tile) for tile in img]239        pixel_values = torch.stack(pixel_values)240        num_patches_list.append(pixel_values.shape[0])241        pixel_values_list.append(pixel_values)242    pixel_values = torch.cat(pixel_values_list)243    return pixel_values, num_patches_list244 245# evaluation setting246max_num_frames = 512247generation_config = dict(248    do_sample=False,249    temperature=0.0,250    max_new_tokens=1024,251    top_p=0.1,252    num_beams=1253)254video_path = "your_video.mp4"255num_segments=128256 257 258with torch.no_grad():259  260  pixel_values, num_patches_list = load_video(video_path, num_segments=num_segments, max_num=1, get_frame_by_duration=False)261  pixel_values = pixel_values.to(torch.bfloat16).to(model.device)262  video_prefix = "".join([f"Frame{i+1}: <image>\n" for i in range(len(num_patches_list))])263  # single-turn conversation264  question1 = "Describe this video in detail."265  question = video_prefix + question1266  output1, chat_history = model.chat(tokenizer, pixel_values, question, generation_config, num_patches_list=num_patches_list, history=None, return_history=True)267  print(output1)268  269  # multi-turn conversation270  question2 = "How many people appear in the video?"271  output2, chat_history = model.chat(tokenizer, pixel_values, question, generation_config, num_patches_list=num_patches_list, history=chat_history, return_history=True)272  273  print(output2)274```275 276## ✏️ Citation277 278```bibtex279 280@article{wang2025internvideo,281  title={InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling},282  author={Wang, Yi and Li, Xinhao and Yan, Ziang and He, Yinan and Yu, Jiashuo and Zeng, Xiangyu and Wang, Chenting and Ma, Changlian and Huang, Haian and Gao, Jianfei and Dou, Min and Chen, Kai and Wang, Wenhai and Qiao, Yu and Wang, Yali and Wang, Limin},283  journal={arXiv preprint arXiv:2501.12386},284  year={2025}285}286```