openbmb/MiniCPM-V

1---2pipeline_tag: visual-question-answering3---4 5## MiniCPM-V6### News7- [2025.01.14] 🔥🔥 We open source [**MiniCPM-o 2.6**](https://huggingface.co/openbmb/MiniCPM-o-2_6), with significant performance improvement over **MiniCPM-V 2.6**, and support real-time speech-to-speech conversation and multimodal live streaming. Try it now.8- [2024.08.06] 🔥 We open-source [**MiniCPM-V 2.6**](https://huggingface.co/openbmb/MiniCPM-V-2_6), which outperforms GPT-4V on single image, multi-image and video understanding. It advances popular features of MiniCPM-Llama3-V 2.5, and can support real-time video understanding on iPad.9- [2024.05.20] 🔥 GPT-4V level multimodal model [**MiniCPM-Llama3-V 2.5**](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5) is out.10- [2024.04.11] 🔥 [**MiniCPM-V 2.0**](https://huggingface.co/openbmb/MiniCPM-V-2) is out.11  12 13**MiniCPM-V** (i.e., OmniLMM-3B) is an efficient version with promising performance for deployment. The model is built based on SigLip-400M and [MiniCPM-2.4B](https://github.com/OpenBMB/MiniCPM/), connected by a perceiver resampler. Notable features of OmniLMM-3B include:14 15- ⚡️ **High Efficiency.** 16 17  MiniCPM-V can be **efficiently deployed on most GPU cards and personal computers**, and **even on end devices such as mobile phones**. In terms of visual encoding, we compress the image representations into 64 tokens via a perceiver resampler, which is significantly fewer than other LMMs based on MLP architecture (typically > 512 tokens). This allows OmniLMM-3B to operate with **much less memory cost and higher speed during inference**.18 19- 🔥 **Promising Performance.** 20 21  MiniCPM-V achieves **state-of-the-art performance** on multiple benchmarks (including MMMU, MME, and MMbech, etc) among models with comparable sizes, surpassing existing LMMs built on Phi-2. It even **achieves comparable or better performance than the 9.6B Qwen-VL-Chat**.22 23- 🙌 **Bilingual Support.** 24 25  MiniCPM-V is **the first end-deployable LMM supporting bilingual multimodal interaction in English and Chinese**. This is achieved by generalizing multimodal capabilities across languages, a technique from the ICLR 2024 spotlight [paper](https://arxiv.org/abs/2308.12038).26 27### Evaluation28 29<div align="center">30 31<table style="margin: 0px auto;">32<thead>33  <tr>34    <th align="left">Model</th>35    <th>Size</th>36    <th>MME</th>37    <th nowrap="nowrap" >MMB dev (en)</th>38    <th nowrap="nowrap" >MMB dev (zh)</th>39    <th nowrap="nowrap" >MMMU val</th>40    <th nowrap="nowrap" >CMMMU val</th>41  </tr>42</thead>43<tbody align="center">44  <tr>45    <td align="left">LLaVA-Phi</td>46    <td align="right">3.0B</td>47    <td>1335</td>48    <td>59.8</td>49    <td>- </td>50    <td>- </td>51    <td>- </td>52  </tr>53  <tr>54    <td nowrap="nowrap" align="left">MobileVLM</td>55    <td align="right">3.0B</td>56    <td>1289</td>57    <td>59.6</td>58    <td>- </td>59    <td>- </td>60    <td>- </td>61  </tr>62  <tr>63    <td nowrap="nowrap" align="left" >Imp-v1</td>64    <td align="right">3B</td>65    <td>1434</td>66    <td>66.5</td>67    <td>- </td>68    <td>- </td>69    <td>- </td>70  </tr>71  <tr>72    <td nowrap="nowrap" align="left" >Qwen-VL-Chat</td>73    <td align="right" >9.6B</td>74    <td>1487</td>75    <td>60.6 </td>76    <td>56.7 </td>77    <td>35.9 </td>78    <td>30.7 </td>79  </tr>80  <tr>81    <td nowrap="nowrap" align="left" >CogVLM</td>82    <td align="right">17.4B </td>83    <td>1438 </td>84    <td>63.7 </td>85    <td>53.8 </td>86    <td>32.1 </td>87    <td>- </td>88  </tr>89  <tr>90    <td nowrap="nowrap" align="left" ><b>MiniCPM-V</b></td>91    <td align="right">3B </td>92    <td>1452 </td>93    <td>67.9 </td>94    <td>65.3 </td>95    <td>37.2 </td>96    <td>32.1 </td>97  </tr>98</tbody>99</table>100 101</div>102 103 104### Examples105<div align="center">106<table>107  <tr>108    <td>109      <p> 110        <img src="assets/Mushroom_en.gif" width="400"/>111      </p>112    </td>113    <td>114      <p> 115        <img src="assets/Snake_en.gif" width="400"/>116      </p>117    </td>118  </tr>119</table>120</div>121 122 123## Demo124Click here to try out the Demo of [MiniCPM-V](http://120.92.209.146:80).125 126## Deployment on Mobile Phone127Currently MiniCPM-V (i.e., OmniLMM-3B) can be deployed on mobile phones with Android and Harmony operating systems. 🚀 Try it out [here](https://github.com/OpenBMB/mlc-MiniCPM).128 129 130## Usage131Inference using Huggingface transformers on Nivdia GPUs or Mac with MPS (Apple silicon or AMD GPUs). Requirements tested on python 3.10：132```133Pillow==10.1.0134timm==0.9.10135torch==2.1.2136torchvision==0.16.2137transformers==4.36.0138sentencepiece==0.1.99139```140 141```python142# test.py143import torch144from PIL import Image145from transformers import AutoModel, AutoTokenizer146 147model = AutoModel.from_pretrained('openbmb/MiniCPM-V', trust_remote_code=True, torch_dtype=torch.bfloat16)148# For Nvidia GPUs support BF16 (like A100, H100, RTX3090)149model = model.to(device='cuda', dtype=torch.bfloat16)150# For Nvidia GPUs do NOT support BF16 (like V100, T4, RTX2080)151#model = model.to(device='cuda', dtype=torch.float16)152# For Mac with MPS (Apple silicon or AMD GPUs).153# Run with `PYTORCH_ENABLE_MPS_FALLBACK=1 python test.py`154#model = model.to(device='mps', dtype=torch.float16)155 156tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V', trust_remote_code=True)157model.eval()158 159image = Image.open('xx.jpg').convert('RGB')160question = 'What is in the image?'161msgs = [{'role': 'user', 'content': question}]162 163res, context, _ = model.chat(164    image=image,165    msgs=msgs,166    context=None,167    tokenizer=tokenizer,168    sampling=True,169    temperature=0.7170)171print(res)172```173 174Please look at [GitHub](https://github.com/OpenBMB/OmniLMM) for more detail about usage.175 176## License177 178#### Model License179* The code in this repo is released under the [Apache-2.0](https://github.com/OpenBMB/MiniCPM/blob/main/LICENSE) License. 180* The usage of MiniCPM-V series model weights must strictly follow [MiniCPM Model License.md](https://github.com/OpenBMB/MiniCPM/blob/main/MiniCPM%20Model%20License.md).181* The models and weights of MiniCPM are completely free for academic research. after filling out a ["questionnaire"](https://modelbest.feishu.cn/share/base/form/shrcnpV5ZT9EJ6xYjh3Kx0J6v8g) for registration, are also available for free commercial use.182 183 184#### Statement185* As a LLM, MiniCPM-V generates contents by learning a large mount of texts, but it cannot comprehend, express personal opinions or make value judgement. Anything generated by MiniCPM-V does not represent the views and positions of the model developers186* We will not be liable for any problems arising from the use of the MinCPM-V open Source model, including but not limited to data security issues, risk of public opinion, or any risks and problems arising from the misdirection, misuse, dissemination or misuse of the model.187 188