Modelpublic

iproskurina/aya-expanse-8b-gptqmodel-4bit

sourceHugging Facecc-by-nc-4.0updated 11mo agoView on Hugging Face

0likes12downloads

Model Card

aya-expanse-8b - GPTQ (4-bit)

Source model: CohereLabs/aya-expanse-8b

This model was quantized to 4-bit using GPTQModel.

Quantization parameters:

—bits: 4
—group_size: 128
—damp_percent: 0.05
—desc_act: False

Usage

            # pip install transformers gptqmodel --no-build-isolation
            from gptqmodel import GPTQModel
            model_id = "iproskurina/aya-expanse-8b-gptqmodel-4bit"
            model = GPTQModel.load(model_id)
            result = model.generate("Uncovering deep insights")[0]
            print(model.tokenizer.decode(result))