OpenGVLab/InternVL-Chat-V1-5-AWQ

1---2license: mit3pipeline_tag: image-text-to-text4library_name: transformers5base_model: OpenGVLab/InternVL-Chat-V1-56new_version: OpenGVLab/InternVL2_5-26B-AWQ7base_model_relation: quantized8language:9  - multilingual10tags:11  - internvl12  - custom_code13---14 15# InternVL-Chat-V1-5-AWQ16 17[\[📂 GitHub\]](https://github.com/OpenGVLab/InternVL)  [\[📜 InternVL 1.0\]](https://huggingface.co/papers/2312.14238)  [\[📜 InternVL 1.5\]](https://huggingface.co/papers/2404.16821)  [\[📜 Mini-InternVL\]](https://arxiv.org/abs/2410.16261)  [\[📜 InternVL 2.5\]](https://huggingface.co/papers/2412.05271)18 19[\[🆕 Blog\]](https://internvl.github.io/blog/)  [\[🗨️ Chat Demo\]](https://internvl.opengvlab.com/)  [\[🤗 HF Demo\]](https://huggingface.co/spaces/OpenGVLab/InternVL)  [\[🚀 Quick Start\]](#quick-start)  [\[📖 Documents\]](https://internvl.readthedocs.io/en/latest/)20 21## Introduction22 23<div align="center">24  <img src="https://raw.githubusercontent.com/InternLM/lmdeploy/0be9e7ab6fe9a066cfb0a09d0e0c8d2e28435e58/resources/lmdeploy-logo.svg" width="450"/>25</div>26 27### INT4 Weight-only Quantization and Deployment (W4A16)28 29LMDeploy adopts [AWQ](https://arxiv.org/abs/2306.00978) algorithm for 4bit weight-only quantization. By developed the high-performance cuda kernel, the 4bit quantized model inference achieves up to 2.4x faster than FP16.30 31LMDeploy supports the following NVIDIA GPU for W4A16 inference:32 33- Turing(sm75): 20 series, T434 35- Ampere(sm80,sm86): 30 series, A10, A16, A30, A10036 37- Ada Lovelace(sm90): 40 series38 39Before proceeding with the quantization and inference, please ensure that lmdeploy is installed.40 41```shell42pip install lmdeploy>=0.5.343```44 45This article comprises the following sections:46 47<!-- toc -->48 49- [Inference](#inference)50- [Service](#service)51 52<!-- tocstop -->53 54### Inference55 56Trying the following codes, you can perform the batched offline inference with the quantized model:57 58```python59from lmdeploy import pipeline, TurbomindEngineConfig60from lmdeploy.vl import load_image61 62model = 'OpenGVLab/InternVL-Chat-V1-5-AWQ'63image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')64backend_config = TurbomindEngineConfig(model_format='awq')65pipe = pipeline(model, backend_config=backend_config, log_level='INFO')66response = pipe(('describe this image', image))67print(response.text)68```69 70For more information about the pipeline parameters, please refer to [here](https://github.com/InternLM/lmdeploy/blob/main/docs/en/inference/pipeline.md).71 72### Service73 74LMDeploy's `api_server` enables models to be easily packed into services with a single command. The provided RESTful APIs are compatible with OpenAI's interfaces. Below are an example of service startup:75 76```shell77lmdeploy serve api_server OpenGVLab/InternVL-Chat-V1-5-AWQ --server-port 23333 --model-format awq78```79 80To use the OpenAI-style interface, you need to install OpenAI:81 82```shell83pip install openai84```85 86Then, use the code below to make the API call:87 88```python89from openai import OpenAI90 91client = OpenAI(api_key='YOUR_API_KEY', base_url='http://0.0.0.0:23333/v1')92model_name = client.models.list().data[0].id93response = client.chat.completions.create(94    model=model_name,95    messages=[{96        'role':97        'user',98        'content': [{99            'type': 'text',100            'text': 'describe this image',101        }, {102            'type': 'image_url',103            'image_url': {104                'url':105                'https://modelscope.oss-cn-beijing.aliyuncs.com/resource/tiger.jpeg',106            },107        }],108    }],109    temperature=0.8,110    top_p=0.8)111print(response)112```113 114## License115 116This project is released under the MIT License. This project uses the pre-trained internlm2-chat-20b as a component, which is licensed under the Apache License 2.0.117 118## Citation119 120If you find this project useful in your research, please consider citing:121 122```BibTeX123@article{chen2024expanding,124  title={Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling},125  author={Chen, Zhe and Wang, Weiyun and Cao, Yue and Liu, Yangzhou and Gao, Zhangwei and Cui, Erfei and Zhu, Jinguo and Ye, Shenglong and Tian, Hao and Liu, Zhaoyang and others},126  journal={arXiv preprint arXiv:2412.05271},127  year={2024}128}129@article{gao2024mini,130  title={Mini-internvl: A flexible-transfer pocket multimodal model with 5\% parameters and 90\% performance},131  author={Gao, Zhangwei and Chen, Zhe and Cui, Erfei and Ren, Yiming and Wang, Weiyun and Zhu, Jinguo and Tian, Hao and Ye, Shenglong and He, Junjun and Zhu, Xizhou and others},132  journal={arXiv preprint arXiv:2410.16261},133  year={2024}134}135@article{chen2024far,136  title={How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites},137  author={Chen, Zhe and Wang, Weiyun and Tian, Hao and Ye, Shenglong and Gao, Zhangwei and Cui, Erfei and Tong, Wenwen and Hu, Kongzhi and Luo, Jiapeng and Ma, Zheng and others},138  journal={arXiv preprint arXiv:2404.16821},139  year={2024}140}141@inproceedings{chen2024internvl,142  title={Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks},143  author={Chen, Zhe and Wu, Jiannan and Wang, Wenhai and Su, Weijie and Chen, Guo and Xing, Sen and Zhong, Muyan and Zhang, Qinglong and Zhu, Xizhou and Lu, Lewei and others},144  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},145  pages={24185--24198},146  year={2024}147}148```149