CoolFace
Modelpublic

iproskurina/Qwen2.5-7B-gptqmodel-4bit

sourceHugging Faceapache-2.0updated 11mo agoView on Hugging Face
0likes15downloads
Model Card

Qwen2.5-7B - GPTQ (4-bit)

Source model: Qwen/Qwen2.5-7B

This model was quantized to 4-bit using GPTQModel.

Quantization parameters:

  • —bits: 4
  • —group_size: 128
  • —damp_percent: 0.05
  • —desc_act: False

Usage

# pip install transformers gptqmodel --no-build-isolation
from gptqmodel import GPTQModel
model_id = "iproskurina/Qwen2.5-7B-gptqmodel-4bit"
model = GPTQModel.load(model_id)
result = model.generate("Uncovering deep insights")[0]
print(model.tokenizer.decode(result))