CoolFace
Modelpublic

drawais/Qwen2.5-32B-Instruct-HQQ-INT4

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face
0likes11downloads
Model Card

Qwen2.5-32B-Instruct-HQQ-INT4

INT4 weight-only quantization of `Qwen/Qwen2.5-32B-Instruct`.

Qwen 2.5 32B-Instruct in HQQ INT4 (alternative to AWQ). Calibration-free quantization.

PropertyValue
Base modelQwen/Qwen2.5-32B-Instruct
QuantizationINT4 weight-only
Approx. on-disk size~20.7 GB
LicenseApache License, Version 2.0
LanguagesEnglish

Load (vLLM)

bash
vllm serve drawais/Qwen2.5-32B-Instruct-HQQ-INT4 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.94
python
from vllm import LLM, SamplingParams
llm = LLM(model="drawais/Qwen2.5-32B-Instruct-HQQ-INT4", max_model_len=32768)
print(llm.generate(["Hello!"], SamplingParams(max_tokens=128))[0].outputs[0].text)

Footprint

~20.7 GB on disk. Recommended VRAM: enough headroom for KV cache.

License & attribution

This artifact is a derivative work of `Qwen/Qwen2.5-32B-Instruct`, released by its original authors under the Apache License, Version 2.0.

This artifact is distributed under the same license. The full license text is included in `LICENSE`, and required attribution is in `NOTICE`.

License text: https://www.apache.org/licenses/LICENSE-2.0 Source model: https://huggingface.co/Qwen/Qwen2.5-32B-Instruct