Aluode/PerceptionLabPortable
0
1o
2 .�Yi.- � @ s� d dl mZ d dlmZ d dlZd dlmZ ddlmZ ddl m3Z4m Z m Z m
Z
ddlmZ ddlmZ dd lmZmZmZmZ e�e�ZG d5d � d ej�ZeG d d
� d
��Z eG dd� d��Z eG dd� d��Z dS )� )�partial)�OptionalN� )�Cache)�BaseModelOutputWithPast� QuestionAnsweringModelOutput� SequenceClassifierOutputWithPast�TokenClassifierOutput)� AutoModel)�Unpack)�TransformersKwargs�auto_docstring�can_return_tuple�loggingc s$ e Zd ZdZdZ� fdd�Z� ZS )�GradientCheckpointingLayera Base class for layers with gradient checkpointing.6 7 This class enables gradient checkpointing functionality for a layer. By default, gradient checkpointing is disabled8 (`gradient_checkpointing = False`). When `model.set_gradient_checkpointing()` is called, gradient checkpointing is9 enabled by setting `gradient_checkpointing = True` and assigning a checkpointing function to `_gradient_checkpointing_func`.10 11 Important:12 13 When using gradient checkpointing with `use_reentrant=True`, inputs that require gradients (e.g. hidden states)14 must be passed as positional arguments (`*args`) rather than keyword arguments to properly propagate gradients.15 16 Example:17 18 ```python19 >>> # Correct - hidden_states passed as positional arg20 >>> out = self.layer(hidden_states, attention_mask=attention_mask)21 22 >>> # Incorrect - hidden_states passed as keyword arg23 >>> out = self.layer(hidden_states=hidden_states, attention_mask=attention_mask)24 ```25 Fc s | j r| jrd}| jj}d|� d�}d|v r$|d r$d|d< |d7 }d}d|v r8|d d ur8d |d< |d7 }d}d |v rL|d d urLd |d < |d267 }d}d |v r`|d d ur`d |d < |d 7 }d}|rn|�d
�d }t�|� | jtt � j27fi |��g|�R � S t � j28|i |��S )NFz7Caching is incompatible with gradient checkpointing in z . Setting� use_cachez `use_cache=False`,TZpast_key_valuez `past_key_value=None`,�past_key_valuesz `past_key_values=None`,Z29layer_pastz `layer_past=None`,�,�.) �gradient_checkpointingZtraining� __class__�__name__�rstrip�logger� warning_onceZ _gradient_checkpointing_funcr �super�__call__)�self�args�kwargsZdo_warnZ30layer_name�message�r � ��E:\DocsHouse\542 percep lab latest\PerceptionLab\PerceptionLab_Portable\python_embed\Lib\site-packages\transformers/modeling_layers.pyr < s2 31"z#GradientCheckpointingLayer.__call__)r �32__module__� __qualname__�__doc__r r �
__classcell__r" r" r! r# r # s r c � e Zd ZdZ� fdd�Zee ddeej deej33 deej dee d eej d34eej d ee
d ee d
efdd���Z� ZS )� GenericForSequenceClassification�modelc sJ t � �|� |j| _t| | jt�|�� tj|j | jdd�| _35| � � d S )NF)Zbias) r �__init__�36num_labels�setattr�base_model_prefixr37 � from_config�nn�Linear� hidden_size�score� post_init�r �configr! r" r# r+ e s38 z)GenericForSequenceClassification.__init__N� input_ids�attention_mask� position_idsr �
inputs_embeds�labelsr r �returnc K s2 t | | j�|f|||||d�|��} | j}39| �|40�} |d ur$|jd } n|jd } | jjd u r7| dkr7td��| jjd u r@d}
n1|d ure|| jjk�| j t41j �}t42j |jd | j t43j d�}|| �
d�}
n d}
t�| jj� d�� | t44j | | j d�|
f }d }|d ur�| j| ||| jd �}t||| j| j| jd45�S ) N�r8 r9 r r: r r r z=Cannot handle batch sizes > 1 if no padding token is defined.�����)�deviceZdtypez� will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`)r? )�logitsr; �
pooled_logitsr6 )�lossr@ r �
hidden_states�46attentions)�getattrr. �last_hidden_stater3 �shaper6 Z pad_token_id�47ValueError�tor? �torchZint32ZarangeZargmaxr r r r �
loss_functionr r rC rD )r r7 r8 r9 r r: r; r r Ztransformer_outputsrC r@ Z48batch_sizeZlast_non_pad_tokenZ non_pad_maskZ
token_indicesrA rB r" r" r# �forwardo sN 49
��� 50 51 ��z(GenericForSequenceClassification.forward�NNNNNNN)r r$ r% r. r+ r r
r rJ �52LongTensor�Tensorr � FloatTensor�boolr r r rL r' r" r" r! r# r) a s> 53�������� �54�r) c s� e Zd ZdZ� fdd�Zdd� Zdd� Zee dd e e55j d56e e57j d e e58j d e e
d
e e59j de e60j de e61j dee defdd���Z� ZS )�GenericForQuestionAnsweringr* c s< t � �|� t| | jt�|�� t�|jd�| _ | �62� d S )N� ) r r+ r- r. r63 r/ r0 r1 r2 �64qa_outputsr4 r5 r! r" r# r+ � s z$GenericForQuestionAnswering.__init__c C s t | | j�jS �N�rE r. Z embed_tokens)r r" r" r# �get_input_embeddings� s z0GenericForQuestionAnswering.get_input_embeddingsc C s |t | | j�_d S rU rV )r �valuer" r" r# �set_input_embeddings� s z0GenericForQuestionAnswering.set_input_embeddingsNr7 r8 r9 r r: �start_positions�
end_positionsr r< c K s� t | | j�|f||||d�|��} | j}65| �|66�} | jddd�\} }
| �d��� } |
�d��� }
d }|d urF|d urF| j| |
||fi |��}t|| |
| j | j67d�S )N)r8 r9 r r: r r> )�dim)rB � start_logits�68end_logitsrC rD ) rE r. rF rT �splitZsqueeze�69contiguousrK r rC rD )r r7 r8 r9 r r: rZ r[ r �outputs�sequence_outputr@ r] r^ rB r" r" r# rL � s2 70
��� 71�z#GenericForQuestionAnswering.forwardrM )r r$ r% r. r+ rW rY r r
r rJ rN rO r rP r r r rL r' r" r" r! r# rR � sB �������� �72�rR c r( )� GenericForTokenClassificationr* c s� t � �|� |j| _t| | jt�|�� t|dd �d ur |j}nt|dd �d ur,|j }nd}t73� |�| _ t74�
|j|j�| _| �� d S )N�classifier_dropout�hidden_dropoutg�������?)r r+ r, r- r. r75 r/ rE rd re r0 ZDropout�dropoutr1 r2 r3 r4 )r r6 rd r! r" r# r+ � s z&GenericForTokenClassification.__init__Nr7 r8 r9 r r: r; r r r< c
K sn t | | j�|f|||||d�|��} | j}76| �|77�}78| �|79�} d } |d ur-| �| || j�} t| | | j| j d�S )Nr= )rB r@ rC rD )80rE r. rF rf r3 rK r6 r rC rD )
r r7 r8 r9 r r: r; r r ra rb r@ rB r" r" r# rL � s. 81
��� 8283�z%GenericForTokenClassification.forwardrM )r r$ r% r. r+ r r
r rJ rN rO r rP rQ r r r rL r' r" r" r! r# rc � s> �������� �84�rc ) � functoolsr �typingr rJ Ztorch.nnr0 Z cache_utilsr Zmodeling_outputsr r r r Z models.autor85 Zprocessing_utilsr �utilsr r
r r Z86get_loggerr r �Moduler r) rR rc r" r" r" r# �<module> s"
87>J<