Aluode/PerceptionLabPortable
0
1o
2 /�Yib- � @ sj d dl mZmZ d dlZd dlmZ ddlmZmZ ddl m3Z4 ddl m Z ddl
mZ ddlmZ dd lmZ dd5lmZmZ dd lmZ d d
lmZ d dlmZm Z m Z m Z m Z m Z m!Z!m"Z" ddl#m$Z$ e�%e&�Z'dZ(dZ)G dd� de�Z*G dd� de�Z+G dd� de �Z,G dd� de �Z-G dd � d e �Z.G d d � d e �Z/G d d � d e �Z0G d!d"� d"e �Z1g d#�Z2dS )$� )�Callable�OptionalN� )�Cache� DynamicCache)�create_causal_mask)�GradientCheckpointingLayer)�BaseModelOutputWithPast)�ALL_ATTENTION_FUNCTIONS)�Unpack)�TransformersKwargs�logging)�deprecate_kwarg� )�CLIPMLP)�LlamaAttention�LlamaForCausalLM� LlamaForSequenceClassification�LlamaForTokenClassification�6LlamaModel�LlamaRotaryEmbedding�apply_rotary_pos_emb�eager_attention_forward� )� PhiConfigzmicrosoft/phi-1r c s� e Zd Zdedef� fdd� Zedddd� dd7ejd e ejejf d e8ej de9e d
e10ej de eje11ej f f dd��Z
� ZS )� PhiAttention�config� layer_idxc s� t � �||� tj|j|j| j dd�| _tj|j|j| j dd�| _ tj|j|j| j dd�| _12tj|j| j |jdd�| _ | ` t
| j|j �| _|j| _| jrotj|j|j |jdd�| _tj|j|j |jdd�| _d S d S )NT�Zbias)�epsZelementwise_affine)�super�__init__�nn�Linear� hidden_sizeZnum_attention_heads�head_dim�q_projZnum_key_value_heads�k_proj�v_proj�denseZo_proj�intZpartial_rotary_factor� rotary_ndims� qk_layernorm� LayerNorm�layer_norm_eps� q_layernorm� k_layernorm��selfr r �� __class__� ��E:\DocsHouse\542 percep lab latest\PerceptionLab\PerceptionLab_Portable\python_embed\Lib\site-packages\transformers/models/phi/modular_phi.pyr! % s � ��zPhiAttention.__init__�past_key_value�past_key_values�4.58��new_name�versionN�
hidden_states�position_embeddings�attention_mask�cache_position�returnc K s� |j d d� }g |�d�| j�R }| �|��|��dd�} | �|��|��dd�}13| �|��|��dd�} | jrB| �| �} | � |14�}15|\} }
| dd | j16�f | d| j17d �f }}|18dd | j19�f |20d| j21d �f }}t ||| |
�\}}t j
||fdd�} t j
||fdd�}22|d ur�|
| |d�}|�|23| | j|�\}24} t}| jjdkr�t| jj }|| | |25| |f| js�dn| j| jd �|��\}}|jg |�d�R � �� }| �|�}||fS )26N�����r r .)�dim)�sin�cosr@ �eagerg )Zdropout�scaling)�shaper% r& �viewZ transposer' r( r, r/ r0 r+ r �torch�cat�updater r r Z_attn_implementationr27 �trainingZattention_dropoutrG Zreshape�28contiguousr) )r2 r= r> r? r8 r@ �kwargsZ input_shapeZ hidden_shapeZ query_statesZ29key_statesZ value_statesrE rD Z query_rotZ30query_passZkey_rotZkey_passZ cache_kwargsZattention_interfaceZ attn_outputZ attn_weightsr5 r5 r6 �forward6 sN 313233�� ��34� 35zPhiAttention.forward)NN)�__name__�36__module__� __qualname__r r* r! r rJ �Tensor�tupler r �37LongTensorrP �
__classcell__r5 r5 r3 r6 r $ s$ �������r c @ � e Zd ZdS )�PhiMLPN�rQ rR rS r5 r5 r5 r6 rY u � rY c s� e Zd Zdedef� fdd� Zedddd� 38 39 dd ejd e ej d
e ej40 de e de e de e de ej41 de e
ejejf de
eje e
ejejf f fdd��Z� ZS )�PhiDecoderLayerr r c sH t � �� t||d�| _t|�| _tj|j|j d�| _42t� |j �| _
d S )N)r �r )r r! r � self_attnrY �mlpr" r- r$ r. �input_layernorm�DropoutZ resid_pdrop�
resid_dropoutr1 r3 r5 r6 r! z s43 4445zPhiDecoderLayer.__init__r7 r8 r9 r: NFr= r? � position_ids�output_attentions� use_cacher@ r> rA c K sr |}46| � |�}| jd||||||||d�| ��\} } | �| �} | �| �|��}
| |
|47 }|f}|r7|| f7 }|S )N)r= r? rc r8 rd re r@ r> r5 )r` r^ rb r_ )r2 r= r? rc r8 rd re r@ r> rO ZresidualZ attn_outputsZself_attn_weightsZfeed_forward_hidden_statesZoutputsr5 r5 r6 rP � s*
48� 49�50 51zPhiDecoderLayer.forward)NNNFFNN)rQ rR rS r r* r! r rJ rT r rV r �boolrU � FloatTensorrP rW r5 r5 r3 r6 r\ y s: �������� � �r\ c @ rX )�PhiRotaryEmbeddingNrZ r5 r5 r5 r6 rh � r[ rh c s� e Zd Zdef� fdd� Z ddeej deej deej dee d eej52 d53ee d ee d ee d
eej de e
defdd�Z� ZS )�PhiModelr c sV t � �� � t�� fdd�t� j�D ��| _t�� j�| _ tj54� j � j d�| _
| `d S )Nc s g | ]}t � |��qS r5 )r\ )�.0r �r r5 r6 �55<listcomp>� s z%PhiModel.__init__.<locals>.<listcomp>r] )r r! r" Z56ModuleList�range�num_hidden_layers�layersra Z57embd_pdrop�
embed_dropoutr- r$ r. �final_layernormZnorm�r2 r r3 rk r6 r! � s �zPhiModel.__init__N� input_idsr? rc r8 �
inputs_embedsre rd �output_hidden_statesr@ rO rA c58 K s� |d ur|n| j j}|d ur|n| j j}|d ur|n| j j}|d u |d uA r*td��| jr9| jr9|r9t�d� d}|d u rB| � |�}|rN|d u rNt59| j d�}| d u rj|d urZ|� � nd} t j
| | |jd |jd�} |d u rs| �d�}t| j ||| ||d�} | �|�}|}
| �|
|�}|r�d nd }|r�d nd }| jd | j j� D ]&}|r�||
f7 }||
f| ||||| |d60�|61��}|d }
|r�||d f7 }q�| �|
�}
|r�||
f7 }t|
|r�|nd ||d �S ) Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.Frk r r )�device)r Z input_embedsr? r@ r8 rc r5 )r? rc r8 rd re r@ r> )Zlast_hidden_stater8 r= Z62attentions)r rd ru re �63ValueErrorZgradient_checkpointingrM �loggerZ warning_onceZ embed_tokensr Zget_seq_lengthrJ ZarangerH rv Z unsqueezer rp Z64rotary_embro rn rq r )r2 rs r? rc r8 rt re rd ru r@ rO Zpast_seen_tokensZ causal_maskr= r> Zall_hidden_statesZall_self_attnsZ
decoder_layerZ
layer_outputsr5 r5 r6 rP � s�
��65 �66�67 68�� � �697071�zPhiModel.forward) NNNNNNNNN)rQ rR rS r r! r rJ rV rT r rg rf r r r rP rW r5 r5 r3 r6 ri � sD �������� �72� � �ri c s e Zd Z� fdd�Z� ZS )�PhiForCausalLMc s& t � �|� tj|j|jdd�| _d S )NTr )r r! r" r# r$ Z73vocab_sizeZlm_headrr r3 r5 r6 r! s zPhiForCausalLM.__init__)rQ rR rS r! rW r5 r5 r3 r6 ry s ry c @ rX )� PhiForSequenceClassificationNrZ r5 r5 r5 r6 rz r[ rz c @ rX )�PhiForTokenClassificationNrZ r5 r5 r5 r6 r{ # r[ r{ )ZPhiPreTrainedModelri ry rz r{ )3�typingr r rJ Ztorch.nnr" Z cache_utilsr r Z
masking_utilsr Zmodeling_layersr Zmodeling_outputsr Zmodeling_utilsr74 Zprocessing_utilsr �utilsr r
Zutils.deprecationr Zclip.modeling_clipr Zllama.modeling_llamar r r r r r r r Zconfiguration_phir Z75get_loggerrQ rx Z_CHECKPOINT_FOR_DOCZ_CONFIG_FOR_DOCr rY r\ rh ri ry rz r{ �__all__r5 r5 r5 r6 �<module> s4 ( 76 77Q1k