Modelpublic

dropbox-dash/Qwen2-VL-2B-Instruct_4bitgs64_hqq_hf

sourceHugging Faceapache-2.0updated 1y agoView on Hugging Face

1likes45downloads

Model Card

This is an <a href="https://github.com/mobiusml/hqq/">HQQ</a> all 4-bit (group-size=64) quantized <a href="https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct"> Qwen2-VL-2B-Instruct</a> model.

Usage

First, install the dependecies:

pip install hqq gemlite; #to use the gemlite backend

Then you can use the sample code below:

Python

import torch
device        = 'cuda:0'
backend       = 'torchao_int4' #'torchao_int4' or 'gemlite'
compute_dtype = torch.bfloat16 if backend=="torchao_int4" else torch.float16
model_id      = 'mobiuslabsgmbh/Qwen2-VL-2B-Instruct_4bitgs64_hqq_hf' 

#Load model
from transformers import AutoModelForCausalLM, AutoProcessor 

model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    torch_dtype=compute_dtype, 
    device_map=device, 
)

processor = AutoProcessor.from_pretrained(model_id)

#Patching
from hqq.utils.patching import prepare_for_inference
prepare_for_inference(model, backend=backend, verbose=True)

Use in <a href="https://github.com/vllm-project/vllm/">vllm</a>:

Python

from vllm import LLM
from vllm.sampling_params import SamplingParams

model_id = "mobiuslabsgmbh/Qwen2-VL-2B-Instruct_4bitgs64_hqq_hf"

llm = LLM(model=model_id, max_model_len=4096, max_num_seqs=2, limit_mm_per_prompt={"image": 1}, dtype=torch.float16)