Modelpublic

modelscope/Yi-1.5-6B-Chat-GPTQ

sourceHugging Faceapache-2.0updated 2y agoView on Hugging Face

0likes20downloads

Model Card

language:

—en
—zh pipeline_tag: text-generation tags:
—gptq
—int4
—yi1.5-6B-Chat
—pytorch license: apache-2.0 licensename: apache-2.0 licenselink: LICENSE ---

About Quantization

我们使用modelscope swift仓库进行GPTQ 4bit量化. 量化文档可以查看这里. 量化命令如下:

We use the modelscope swift repository to perform GPTQ 4bit quantization. Quantization documentation can be found here. The quantization command is as follows:

bash

# Experimental Environment: A100
swift export \
    --quant_bits 4 \
    --model_type yi-1_5-6b-chat \
    --quant_method gptq \
    --dataset alpaca-zh alpaca-en sharegpt-gpt4-mini \
    --quant_seqlen 4096

Inference:

bash

CUDA_VISIBLE_DEVICES=0 swift infer --model_type yi-1_5-6b-chat-gptq-int4

SFT:

bash

CUDA_VISIBLE_DEVICES=0 swift sft --model_type yi-1_5-6b-chat-gptq-int4 --dataset leetcode-python-en

Original Model:

YI1.5-6B-Chat