Apppublic
Aluode/PerceptionLabPortable

sourceHugging Faceupdated 9mo agoView on Hugging Face
0likes
rootpython_embedLibsite-packagestransformersmodelsglm__pycache__modular_glm.cpython-310.pyc
modular_glm.cpython-310.pyc35 linesDownload Raw Back to __pycache__
1o

2/�Yi��@s�ddlmZddlZddlmZddlmZddlmZm	Z	m3Z4mZddlm
Z
dd	lmZe�e�Zd5ZGdd�de
�Zd
d�Zddd�ZGdd�de�ZGdd�de	�ZGdd�de6�ZGdd�de�Zgd�ZdS)�)�OptionalN�)�logging�)�LlamaAttention�LlamaForCausalLM�LlamaForSequenceClassification�LlamaForTokenClassification)�Phi3MLP�)�	GlmConfigzTHUDM/glm-4-9bc@�eZdZdS)�GlmMLPN��__name__�7__module__�__qualname__�rr��E:\DocsHouse\542 percep lab latest\PerceptionLab\PerceptionLab_Portable\python_embed\Lib\site-packages\transformers/models/glm/modular_glm.pyr%�rcCs>|dddd�f}|dddd�f}tj||fdd��d�S)	z*Rotates half the hidden dims of the input..rNrr�������dim�����)�torch�stack�flatten)�x�x1Zx2rrr�rotate_half)src
Cs�|�|�}|�|�}|dd|jdd�fjddd�}|dd|jdd�fjddd�}|jd}|dd|�f|d|d�f}}|dd|�f|d|d�f}	}8||t|�|}|	|t|	�|}tj||gdd�}tj||9gdd�}||fS)a�Applies Rotary Position Embedding to the query and key tensors.10 11    Args:12        q (`torch.Tensor`): The query tensor.13        k (`torch.Tensor`): The key tensor.14        cos (`torch.Tensor`): The cosine part of the rotary embedding.15        sin (`torch.Tensor`): The sine part of the rotary embedding.16        position_ids (`torch.Tensor`, *optional*):17            Deprecated and unused.18        unsqueeze_dim (`int`, *optional*, defaults to 1):19            The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and20            sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note21            that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and22            k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes23            cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have24            the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2.25    Returns:26        `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding.27    .Nrrr)Z	unsqueeze�shapeZrepeat_interleaverr�cat)
�q�k�cos�sinZposition_idsZ
unsqueeze_dimZ28rotary_dimZq_rotZq_passZk_rotZk_passZq_embedZk_embedrrr�apply_rotary_pos_emb0s2930$$31""r&cs,eZdZddedeef�fdd�
Z�ZS)�GlmAttentionN�config�	layer_idxcs.t��||�tj|j|j|jdd�|_dS)NF)Zbias)�super�__init__�nnZLinearZnum_attention_headsZhead_dimZhidden_sizeZo_proj)�selfr(r)��	__class__rrr+[s zGlmAttention.__init__)N)rrrrr�intr+�
__classcell__rrr.rr'Zs$r'c@r
)�GlmForCausalLMNrrrrrr2`rr2c@r
)�GlmForSequenceClassificationNrrrrrr3drr3c@r
)�GlmForTokenClassificationNrrrrrr4hrr4)ZGlmPreTrainedModelZGlmModelr2r3r4)Nr)�typingrrZtorch.nnr,�utilsrZllama.modeling_llamarrrr	Zphi3.modeling_phi3r32Zconfiguration_glmrZ33get_loggerr�loggerZ_CHECKPOINT_FOR_DOCrrr&r'r2r3r4�__all__rrrr�<module>s"3435*