Aluode/PerceptionLabPortable
0
1o
2 .�Yi�~ � @ sj d Z ddlmZmZ ddlZddlZddlmZ ddlm Z m3Z4m Z ddl m
Z
mZ ddlmZ dd lmZmZmZ dd5lmZ dd lmZmZmZ dd lm Z m Z d
dl m Z e � e!�Z"dd� Z#dd� Z$d'dd�Z%G dd� dej&�Z'dd� Z(G dd� dej&�Z)e G dd � d e��Z*e G d d � d e*��Z+e d d �G d!d"� d"e*e��Z,e d#d �G d$d%� d%e*��Z-g d&�Z.dS )(zPyTorch CTRL model.� )�Optional�UnionN)�nn)�BCEWithLogitsLoss�CrossEntropyLoss�MSELoss� )�Cache� DynamicCache)�GenerationMixin)�BaseModelOutputWithPast�CausalLMOutputWithPast�SequenceClassifierOutput)�PreTrainedModel)�Conv1D� find_pruneable_heads_and_indices�prune_linear_layer)�auto_docstring�logging� )�6CTRLConfigc C s$ dt �dd|d | � }| | S )Nr i' � )�torch�pow)�pos�i� d_model_sizeZ angle_rates� r ��E:\DocsHouse\542 percep lab latest\PerceptionLab\PerceptionLab_Portable\python_embed\Lib\site-packages\transformers/models/ctrl/modeling_ctrl.py�7angle_defn( s r c C s� t tj| tjd��|��d�tj|tjd��|��d�|�}t�|d d �dd d�f �}t�|d d �dd d�f �}tj||gdd�}|S )N��dtyper r r �������dim) r r �arangeZint64�to� unsqueeze�sin�cos�cat)�positionr r! Z8angle_radsZsinesZcosines� pos_encodingr r r �positional_encoding- s � r- c
C s� t �| |�dddd��}|jd }|t�|� }|d ur7|�d�|�d�} }9|||10| |11�d |12�f d 7 }|d ur?|| }t j|dd�} |d urN| | } t �| |�} | | fS ) Nr r r r r" �����g ���r# )r �matmul�permute�shape�np�sqrt�sizeZsoftmax)
�q�k�v�mask�attention_mask� head_maskZ matmul_qkZdkZscaled_attention_logits�nd�nsZattention_weights�outputr r r � scaled_dot_product_attention< s 13 r> c sD e Zd Zd � fdd� Zdd� Zdd� Z d d d14�Z� ZS )
�MultiHeadAttentionNc sp t � �� || _|| _|| _t|| j �| _t�||�| _ t�||�| _15t�||�| _ t�||�| _ t
� | _d S �N)�super�__init__� num_headsr � layer_idx�int�depthr �Linear�Wq�Wk�Wv�dense�set� pruned_heads)�selfr rC rD �� __class__r r rB W s 16 zMultiHeadAttention.__init__c C s� | j | j }t|�dkrd S t|| j|| j�\}}t| j|�| _t| j|�| _t| j|�| _t| j |dd�| _ | jt|� | _|| j | _ | j�17|�| _d S )Nr r r# ) r rC �lenr rM r rH rI rJ rK �union)rN �headsZattention_head_size�indexr r r � prune_headsf s z MultiHeadAttention.prune_headsc C s" |� |d| j| j�}|�g d��S )Nr" �r r r r )�reshaperC rF r0 )rN �x�18batch_sizer r r �split_into_headsw s z#MultiHeadAttention.split_into_headsFc C s� |j d } | �|�}| �|�}| �|�}| �|| �}| �|| �}| �|| �}|d ur7|�||| jd|19i�\}}t||||||�} | d �g d��}
| d }|
� | d| j20�}| � |�} | |fS )Nr �cache_positionrV r r" ) r1 rH rI rJ rZ �updaterD r> r0 rW r rK )rN r7 r6 r5 r8 �21layer_pastr9 r: � use_cache�output_attentionsr[ rY r= Zscaled_attentionZattnZoriginal_size_attentionr r r �forward{ s 22
232425 26zMultiHeadAttention.forwardr@ �NNNFFN)�__name__�27__module__� __qualname__rB rU rZ r` �
__classcell__r r rO r r? V s 28�r? c C s" t �t �| |�t �� t �|| ��S r@ )r Z29SequentialrG ZReLU)r �dffr r r � point_wise_feed_forward_network� s "rg c s4 e Zd Zd� fdd� Z d dd�Z� ZS )30� EncoderLayer皙�����?Nc sb t � �� t|||d�| _t||�| _tj|dd�| _tj|dd�| _ t�31|�| _ t�32|�| _ d S )N�rD g�����ư>��eps)
rA rB r? �multi_head_attentionrg �ffnr � LayerNorm�33layernorm1�34layernorm2�Dropout�dropout1�dropout2)rN r rC rf ZraterD rO r r rB � s 35 zEncoderLayer.__init__Fc C s~ | � |�} | j| | | |||||||d�36}37|38d } | �| �} || } | �| �}
| �|
�}| �|�}| | }
|
f|39dd � }|S )N�r] r9 r: r^ r_ r[ r r )rp rm rs rq rn rt )rN rX r8 r] r9 r: r^ r_ r[ ZnormedZ attn_outputsZ attn_outputZout1Zout2Z40ffn_output�outputsr r r r` � s, 41 � 42434445zEncoderLayer.forward)ri Nra )rb rc rd rB r` re r r rO r rh � s �rh c @ s" e Zd ZU eed< dZdd� ZdS )�CTRLPreTrainedModel�config� transformerc C s� t |tjtf�r"|jjjd| jjd� |j dur |j j�46� dS dS t |tj �rE|jjjd| jjd� |j durC|jj|j �47� dS dS t |tj
�rZ|j j�48� |jj�d� dS dS )zInitialize the weights.g )�meanZstdN� �?)�49isinstancer rG r �weight�dataZnormal_rx Zinitializer_range�biasZzero_� EmbeddingZ padding_idxro Zfill_)rN �moduler r r �
_init_weights� s 50� 51� �z!CTRLPreTrainedModel._init_weightsN)rb rc rd r �__annotations__Zbase_model_prefixr� r r r r rw � s 52 rw c s� e Zd Z� fdd�Zdd� Zdd� Zdd� Ze dd53ee j54 d ee d ee j d
ee j55 dee j56 dee j dee j dee
dee
dee
dee
dee j deee j ef fdd��Z� ZS )� CTRLModelc s� t � �� � � j| _� j| _t� j| jtj �| _57t � � j
� j�| _t �� j�| _t �� fdd�t� j�D ��| _t j� j� jd�| _| �� d S )Nc s& g | ]}t � j� j� j� j|d ��qS )rj )rh �n_embdZn_headrf Z resid_pdrop)�.0r �rx r r �58<listcomp>� s ��z&CTRLModel.__init__.<locals>.<listcomp>rk )rA rB r� r �n_layerZ59num_layersr- Z n_positionsr �floatr, r r� �60vocab_size�wrr Z61embd_pdrop�dropoutZ62ModuleList�range�hro Zlayer_norm_epsilon� layernorm� post_init�rN rx rO r� r rB � s 63�� zCTRLModel.__init__c C s | j S r@ �r� )rN r r r �get_input_embeddings s z CTRLModel.get_input_embeddingsc C s64 || _ d S r@ r� )rN Znew_embeddingsr r r �set_input_embeddings s 65z CTRLModel.set_input_embeddingsc C s( |� � D ]
\}}| j| j�|� qdS )zv66 Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer}67 N)�itemsr� rm rU )rN Zheads_to_prune�layerrS r r r � _prune_heads s �zCTRLModel._prune_headsN� input_ids�past_key_valuesr9 �token_type_ids� position_idsr: �
inputs_embedsr^ r_ �output_hidden_states� return_dictr[ �returnc
K sD | dur| n| j j} |dur|n| j j}|68dur|69n| j j}70| dur$| n| j j} |dur4|dur4td��|durP| �||� |�� }|�d|d �}|j d }n|durb|�� dd� }|j d }ntd��|durm|j71n|j72}|r||du r|t | j d�}|r�t |t
�r�t�d� t �|�}|dur�|�� nd}|du r�tj||d | tj|d�}|�d�}|dur�|dkr�td ��|�|d�}|�d73��d �}|j| jd �}d
| t�| j�j }| �|| j j�}|dur�|�d|d �}| � |�}|t � | j �9 }nd}|du �r| � |�}|d }t� t�!|| || �d74��|�}|t � | j �9 }| j"�|�| _"| j"|dd�f }|| | }| �#|�}|75�rEdnd}| �rLdnd}t$| j%�D ])\}}|76�r_||f }||||||| || | d�}|d }| �r{||d77 f7 }�qS| �&|�}|78�r�||f }| �s�t
dd� ||||fD ��S t'||||d�S )aE 79 input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`):80 `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]`81 (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary.82 83 If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as84 `input_ids`.85 86 Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and87 [`PreTrainedTokenizer.encode`] for details.88 89 [What are input IDs?](../glossary#input-ids)90 91 Example:92 93 ```python94 >>> from transformers import AutoTokenizer, CTRLModel95 >>> import torch96 97 >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl")98 >>> model = CTRLModel.from_pretrained("Salesforce/ctrl")99 100 >>> # CTRL was trained with control codes as the first token101 >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt")102 >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values()103 104 >>> outputs = model(**inputs)105 106 >>> last_hidden_states = outputs.last_hidden_state107 >>> list(last_hidden_states.shape)108 [1, 5, 1280]109 ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer" r z5You have to specify either input_ids or inputs_embedsr� z�Passing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `DynamicCache` instead, e.g. `past_key_values=DynamicCache.from_legacy_cache(past_key_values)`.)r! �devicez$batch_size has to be defined and > 0r r r r{ r ru c s s � | ] }|d ur|V qd S r@ r )r� r7 r r r � <genexpr>� s � �z$CTRLModel.forward.<locals>.<genexpr>)Zlast_hidden_stater� �
hidden_states�110attentions)(rx r_ r^ r� �use_return_dict�111ValueErrorZ%warn_if_padding_and_no_attention_maskr4 �viewr1 r� r112 r| �tuple�logger� warning_onceZfrom_legacy_cache�get_seq_lengthr r% �longr' r& r! Zfinfo�minZ
get_head_maskr� r� r2 r3 r ZtriuZonesr, r� � enumerater� r� r ) rN r� r� r9 r� r� r: r� r^ r_ r� r� r[ �kwargsZ input_shaperY r� � past_lengthZtoken_type_embedsZseq_lenr8 Z113pos_embedsr� Zall_hidden_statesZall_attentionsr r� rv r r r r` s� 1� �114 115 116117118" 119120�121�122123124��zCTRLModel.forward� NNNNNNNNNNNN)rb rc rd rB r� r� r� r r r �125LongTensorr � FloatTensor�bool�Tensorr r� r r` re r r rO r r� � sZ �������� �126� � �
��r� z�127 The CTRL Model transformer with a language modeling head on top (linear layer with weights tied to the input128 embeddings).129 )Z custom_introc s� e Zd ZdgZ� fdd�Ze ddeej dee deej130 deej d eej d131eej132 d eej133 d eej d
ee dee dee dee deej de
eej ef f dd��Zddd�Z� ZS )�CTRLLMHeadModelzlm_head.weightc s8 t � �|� t|�| _tj|j|jdd�| _| � � d S )NT�r )134rA rB r� ry r rG r� r� �lm_headr� r� rO r r rB � s 135 zCTRLLMHeadModel.__init__Nr� r� r9 r� r� r: r� �labelsr^ r_ r� r� r[ r� c K s� | dur| n| j j} | j|||||||| |136| | |
d� }|d }| �|�}d}|dur8| j||fd| j ji|��}| sN|f|dd� }|durL|f| S |S t|||j|j|j d�S )a137 138 input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`):139 `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]`140 (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary.141 142 If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as143 `input_ids`.144 145 Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and146 [`PreTrainedTokenizer.encode`] for details.147 148 [What are input IDs?](../glossary#input-ids)149 labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):150 Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set151 `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100`152 are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]`153 154 Example:155 156 ```python157 >>> import torch158 >>> from transformers import AutoTokenizer, CTRLLMHeadModel159 160 >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl")161 >>> model = CTRLLMHeadModel.from_pretrained("Salesforce/ctrl")162 163 >>> # CTRL was trained with control codes as the first token164 >>> inputs = tokenizer("Wikipedia The llama is", return_tensors="pt")165 >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values()166 167 >>> sequence_ids = model.generate(inputs["input_ids"])168 >>> sequences = tokenizer.batch_decode(sequence_ids)169 >>> sequences170 ['Wikipedia The llama is a member of the family Bovidae. It is native to the Andes of Peru,']171 172 >>> outputs = model(**inputs, labels=inputs["input_ids"])173 >>> round(outputs.loss.item(), 2)174 9.21175 176 >>> list(outputs.logits.shape)177 [1, 5, 246534]178 ```N) r� r9 r� r� r: r� r^ r_ r� r� r[ r r� r )�loss�logitsr� r� r� )179rx r� ry r� Z
loss_functionr� r
r� r� r� )rN r� r� r9 r� r� r: r� r� r^ r_ r� r� r[ r� �transformer_outputsr� Z lm_logitsr� r= r r r r` � sJ <�180����zCTRLLMHeadModel.forwardc 181 K s� |d ur#|� � }|jd |kr|}n|jd d }|d d �|d �f }|||d�}|�dd � |�� D ]\}} ||vrGtd|� d�� | ||< q3|S )Nr )r� r� r^ r� z Warning: z is not a recognized input.)r� r1 �popr� �print)182rN r� r� r^ r� r� Zremove_prefix_lengthZ model_inputs�key�valuer r r � prepare_inputs_for_generation- s �z-CTRLLMHeadModel.prepare_inputs_for_generation)
NNNNNNNNNNNNN�NN)rb rc rd Z_tied_weights_keysrB r r r r� r r� r� r� r r� r
r` r� re r r rO r r� � s^ �������� �183� � �
�� �er� a� 184 The CTRL Model transformer with a sequence classification head on top (linear layer).185 [`CTRLForSequenceClassification`] uses the last token in order to do the classification, as other causal models186 (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last187 token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in188 each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot189 guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last190 value in each row of the batch).191 c s� e Zd Z� fdd�Ze ddeej dee deej deej deej d eej d192eej d eej d ee193 d
ee194 dee195 dee196 de e ej
ef fdd��Z� ZS )� CTRLForSequenceClassificationc s@ t � �|� |j| _t|�| _tj|j| jdd�| _| � � d S )NFr� )197rA rB �198num_labelsr� ry r rG r� �199classifierr� r� rO r r rB V s200 201 z&CTRLForSequenceClassification.__init__Nr� r� r9 r� r� r: r� r� r^ r_ r� r� r� c
C sJ | dur| n| j j} | j|||||||| |202| | d� }
|
d }| �|�}|dur1|jdd� \}}n |jdd� \}}| j jdu rH|dkrHtd��| j jdu rQd}n1|durv|| j jk�|jt j203�}t j |jd |jt j204d�}|| � d�}n d}t
�| jj� d �� |t j ||jd205�|f }d}|du�r| j jdu r�| jdkr�d | j _n| jdkr�|jt jks�|jt jkr�d | j _nd
| j _| j jd kr�t� }| jdkr�||�� |�� �}n,|||�}n&| j jd kr�t� }||�d| j�|�d��}n| j jd
k�rt� }|||�}| �s|f|
dd� }|du�r|f| S |S t|||
j |
j d�S )a2 206 input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`):207 `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]`208 (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary.209 210 If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as211 `input_ids`.212 213 Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and214 [`PreTrainedTokenizer.encode`] for details.215 216 [What are input IDs?](../glossary#input-ids)217 labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):218 Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,219 config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If220 `config.num_labels > 1` a classification loss is computed (Cross-Entropy).221 222 Example of single-label classification:223 224 ```python225 >>> import torch226 >>> from transformers import AutoTokenizer, CTRLForSequenceClassification227 228 >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl")229 >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl")230 231 >>> # CTRL was trained with control codes as the first token232 >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt")233 >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values()234 235 >>> with torch.no_grad():236 ... logits = model(**inputs).logits237 238 >>> predicted_class_id = logits.argmax().item()239 >>> model.config.id2label[predicted_class_id]240 'LABEL_0'241 ```242 243 ```python244 >>> import torch245 246 >>> torch.manual_seed(42) # doctest: +IGNORE_RESULT247 >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)`248 >>> num_labels = len(model.config.id2label)249 >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels)250 251 >>> labels = torch.tensor(1)252 >>> loss = model(**inputs, labels=labels).loss253 >>> round(loss.item(), 2)254 0.93255 ```256 257 Example of multi-label classification:258 259 ```python260 >>> import torch261 >>> from transformers import AutoTokenizer, CTRLForSequenceClassification262 263 >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl")264 >>> model = CTRLForSequenceClassification.from_pretrained(265 ... "Salesforce/ctrl", problem_type="multi_label_classification"266 ... )267 268 >>> # CTRL was trained with control codes as the first token269 >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt")270 >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values()271 272 >>> with torch.no_grad():273 ... logits = model(**inputs).logits274 275 >>> predicted_class_id = logits.argmax().item()276 >>> model.config.id2label[predicted_class_id]277 'LABEL_0'278 ```279 280 ```python281 >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)`282 >>> num_labels = len(model.config.id2label)283 >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels)284 285 >>> num_labels = len(model.config.id2label)286 >>> labels = torch.nn.functional.one_hot(torch.tensor([predicted_class_id]), num_classes=num_labels).to(287 ... torch.float288 ... )289 >>> loss = model(**inputs, labels=labels).loss290 >>> loss.backward() # doctest: +IGNORE_RESULT291 ```N)292r� r9 r� r� r: r� r^ r_ r� r� r r r z=Cannot handle batch sizes > 1 if no padding token is defined.r" )r� r! z� will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`)r� Z293regressionZsingle_label_classificationZmulti_label_classification)r� r� r� r� ) rx r� ry r� r1 Z pad_token_idr� r& r� r Zint32r% Zargmaxr� r� rP rb Z problem_typer� r! r� rE r Zsqueezer r� r r r� r� )rN r� r� r9 r� r� r: r� r� r^ r_ r� r� r� r� r� rY Zsequence_lengthZlast_non_pad_tokenZ non_pad_maskZ
token_indicesZ
pooled_logitsr� Zloss_fctr= r r r r` _ sx h�294 �295 296297"298 299 300�z%CTRLForSequenceClassification.forwardr� )rb rc rd rB r r r r� r r� r� r r� r� r r` re r r rO r r� J sT �������� �301� � �
��r� )r� r� r� rw r� )/�__doc__�typingr r �numpyr2 r r Ztorch.nnr r r Z cache_utilsr r302 Z303generationr Zmodeling_outputsr r
r Zmodeling_utilsr Z
pytorch_utilsr r r �utilsr r Zconfiguration_ctrlr Z304get_loggerrb r� r r- r> �Moduler? rg rh rw r� r� r� �__all__r r r r �<module> sH 305306G2 M� � ?