Aluode/PerceptionLabPortable
0
1o
2 0�Yiz � @ s| d dl mZmZ ddlmZ erddlmZ ddlmZm Z m3Z4m Z ddl m
Z
e5� r/d dlZe �e�ZG d d6� d7e�ZdS ) � )�
TYPE_CHECKING�Optional� )� HfQuantizer� )�PreTrainedModel)�is_accelerate_available�is_eetq_available�is_torch_available�logging)�get_module_from_nameNc s� e Zd ZdZdZdZddgZ� fdd�Zdd � Zd$d
d�Z ddde8d e fdd�Z ddddde9ddfdd�Z
d%dd�Z d&ddd eee10 fd d �Zd&d d!�Zed e fd"d#��Z� ZS )'�EetqHfQuantizera 11 8-bit quantization from EETQ quantization method:12 before loading: converts transformer layers into W8A16Linear during loading: load 16bit weight and pass to the13 layer object after: quantizes individual weights in Linear8bitLt into 8bit at first .cuda() call14 TF�eetqZ15acceleratec s t � j|fi |�� || _d S �N)�super�__init__�quantization_config)�selfr �kwargs�� __class__� ��E:\DocsHouse\542 percep lab latest\PerceptionLab\PerceptionLab_Portable\python_embed\Lib\site-packages\transformers/quantizers/quantizer_eetq.pyr - s 16zEetqHfQuantizer.__init__c 17 O s� t � std��zdd l}W n ty% } z dt|�v r td�|�� d }~ww t� s-td��|�dd�s9|�dd�r=td ��tj� � sFt18d19��|�d �}|d u rVt � d � d S |d urot
|t�rqd
|�� v skd|�� v rstd��d S d S d S )Nz�Using `eetq` 8-bit quantization requires eetq.Please install the latest version of eetq from : https://github.com/NetEase-FuXi/EETQr Zshard_checkpointz�You are using a version of EETQ that is incompatible with the current transformers version. Either downgrade transformers to <= v4.46.3 or, if available, upgrade EETQ to > v1.0.0.zNLoading an EETQ quantized model requires accelerate (`pip install accelerate`)Zfrom_tfFZ from_flaxz�Converting into 8-bit weights from tf/flax weights is currently not supported, please make sure the weights are in PyTorch format.z/No GPU found. A GPU is needed for quantization.�20device_mapz�You have loaded an EETQ model on CPU and have a CUDA device available, make sure to set your model on a GPU device in order to run your model.�cpuZdiskz�You are attempting to load an EETQ model with a device_map that contains a CPU or disk device. This is not supported. Please remove the CPU or disk device from the device_map.)r � ImportErrorr �strr �get�21ValueError�torch�cudaZ is_available� RuntimeError�loggerZ warning_once�22isinstance�dict�values)r �argsr r �excr r r r �validate_environment1 sH � ���� �2324�"��z$EetqHfQuantizer.validate_environment�dtype� torch.dtype�returnc C s6 |d u rt j}t�d|� |S |t jkrt�d� |S )Nz�Overriding dtype=%s with `dtype=torch.float16` due to requirements of `eetq` to enable model loading in 8-bit. Pass your own dtype to specify the dtype of the remaining non-linear layers or pass dtype=torch.float16 to remove this warning.zLWe suggest you to set `dtype=torch.float16` for better efficiency with EETQ.)r Zfloat16r" �info)r r) r r r � update_dtype_ s � 25�26z EetqHfQuantizer.update_dtype�modelr �27param_namec K s> ddl m} t||�\}}t||�r | js|dkrdS dS dS )Nr )�28EetqLinear�biasFT)r r0 r r# �
pre_quantized)r r. r/ r r0 �module� tensor_namer r r �param_needs_quantizationm s 29z(EetqHfQuantizer.param_needs_quantization� param_valuez torch.Tensor�
target_devicez torch.devicec K s� ddl m}m} t||�\}} ||�\}30} t||�r8| js!| dkr0| dkr/|jtjkr/t d��n| dkr8t d��|31�32|�|j | < |� d| �33|�� d S ) Nr )r0 � quantize_and_preprocess_weightsr1 �weightz6Expect quantized weights but got an unquantized weight� weight_scalez;Expect unquantized weights but got a quantized weight_scaleZ
weight_scales)
r r0 r8 r r# r2 r) r Zint8r �to�_buffers�register) r r. r6 r/ r7 r r0 r8 r3 r4 � new_valuer: r r r �create_quantized_paramy s 34�z&EetqHfQuantizer.create_quantized_paramc K s |S r r )r r. r r r r �#_process_model_after_weight_loading� � z3EetqHfQuantizer._process_model_after_weight_loadingN�keep_in_fp32_modulesc K sD ddl m} | �|| jj|�| _||| j| j| jd�}| j|j_d S )Nr )�replace_with_eetq_linear)�modules_to_not_convertr r2 )Z integrationsrC Zget_modules_to_not_convertr rD r2 �config)r r. rB r rC r r r �$_process_model_before_weight_loading� s 35��z4EetqHfQuantizer._process_model_before_weight_loadingc C � dS �NTr )r Zsafe_serializationr r r �is_serializable� rA z EetqHfQuantizer.is_serializablec C rG rH r )r r r r � is_trainable� s z EetqHfQuantizer.is_trainable)r) r* r+ r* )r. r r )�__name__�36__module__� __qualname__�__doc__Z requires_parameters_quantizationZrequires_calibrationZrequired_packagesr r( r- r �boolr5 r? r@ r �listrF rI �propertyrJ �
__classcell__r r r r r
! s8 37. ���38�39��4041�42r
)�typingr r �baser Zmodeling_utilsr �utilsr r r43 r Zquantizers_utilsr r Z44get_loggerrK r" r
r r r r �<module> s
45