msj19/gated_deltaproduct
05
1�
2 � h� � �� � d Z ddlmZ ddlmZ ddlZddlmZ ddlm Z ddl3m Z ddl m
Z
mZ ddlmZ G d � d4ej" � Zed k( �r&d Zd
Zd
Zej. Z ej2 eee� j5 e� j7 � j9 d� Z ej2 eee� j5 e� j7 � Z eed�� j5 e� j7 � Z e e � Z e jC e d�� e jD dcZ#e _" e$e jJ � de _% e e � Z& e$e jJ � e&jC e � e$d� yy)zb5https://github.com/corl-team/rebased/blob/main/flash_linear_attention/fla/layers/rebased_fast.py6� )� annotations)�OptionalN)� rearrange)�RebasedFeatureMap)�chunk_linear_attn�fused_chunk_linear_attn)�parallel_rebasedc � � � e Zd Z d d� fd�
Zdd�Zdd�Zd d7d�Z� xZS ) �ReBasedLinearAttentionc
� �� t �| � � || _ || _ | | _ | j dv sJ �|| _ || _ || _ | j | j z | _ || _ || _8 || _ | | _ t | j9 |||� | _ t j | j | j10 | j z d�� | _ t j | j | j11 | j z d�� | _ t j | j | j | j z d�� | _ t j | j | j z | j d�� | _ t j* � | _ |12| _ | j1 | j2 � y )N)� fused_chunk�parallel�chunkF)�bias)�super�__init__� hidden_size�l_max�mode� feature_dim�num_key_value_heads� num_heads�head_dim� use_gamma�use_beta� normalize�causalr � feature_map�nn�Linear�q_proj�k_proj�v_proj�o_proj�Identity�dropout�eps�apply�_initialize_weights)�selfr r r r r r r r r r' r � layer_idx�kwargs� __class__s ��4/9950backfile/msj/new_gen_test/fla/layers/rebased.pyr z ReBasedLinearAttention.__init__ sl �� � ����&�����13��� ��y�y� @�@�@�@� '���#6�� �"��� �(�(�D�,D�,D�D��
�"��� ��
�"��� �� �,�T�-=�-=�y�(�T]�^����i�i�� 0� 0�$�2B�2B�T�^�^�2S�Z_�`�� ��i�i�� 0� 0�$�2B�2B�T�^�^�2S�Z_�`�� ��i�i�� 0� 0�$�2J�2J�T�]�]�2Z�af�g�� ��i�i����� � � >��@P�@P�W\�]�� ��{�{�}�� ���� �14�15�4�+�+�,� c �" � t |dd� ry t |t j � r`t j j |j d�� |j �)t j j |j � d|_ y )N�_is_hf_initializedFg�;f���?)�gainT)16�getattr�17isinstancer r �init�xavier_uniform_�weightr �zeros_r1 )r* �modules r. r) z*ReBasedLinearAttention._initialize_weights? sa � � �6�/�� 7� � �f�b�i�i� (� �G�G� #� #�F�M�M� � #� B��{�{�&������v�{�{�+�$(��!r/ c �$ � � � j }� j |� � j |� � j |� }}}t � fd�|||g� \ }}}� j ||dk7 �� � j ||dk7 �� }}|dk( rt
|||dd�� }nI|dk( rt |||dd�� }n3|dk( r.|j d d18k sJ �t |||� j dd� }t d � }� j |� }� j |� }|S ) Nc �4 �� t | d�j �� S )Nzb l (h d) -> b h l d)�h)r r )�xr* s �r. �<lambda>z0ReBasedLinearAttention.forward.<locals>.<lambda>K s �� � �!�-C�t�~�~� Vr/ r )�flattenr
T� )r �scaler ������ �b h l d -> b l (h d))r r! r" r# �mapr r r �shaper r' r r$ r& )r* �
hidden_statesr, r �q�k�v�os ` r. �forwardz ReBasedLinearAttention.forwardH s �� ��y�y���+�+�m�,�d�k�k�-�.H�$�+�+�Vc�Jd�a�1���V�YZ�\]�_`�Xa�b���1�a�� � ��D�J�,>� �@�$�BR�BR�ST�_c�gq�_q�BR�Bs�1�� �=� �'��1�a�4�q�I�A�
�W�_�!�!�Q��T��C�A�
�Z�
��7�7�2�;�#�%� %�%� ��A�q�$�(�(�D�$�?�A� �a�/� 0�� �K�K��N�� �L�L��O���r/ c �� � |j � \ }}}| j |� | j |� | j |� }19} }|j ||| j20 | j � j dd� }| j ||| j | j � j dd� } |21j ||| j | j � j dd� }22| j |� | j | � } }|j d� | j d� |23j d� }24} }| j rX|| |25z j d� z j d� || j d� z j d� | j z z } nY|| |26z j dd� z j d� || j dd� z j d� | j z z } t! | d� } | j# | j% |j& � � } | j) | � } | j% |j& � S )zq27 x (torch.Tensor): tensor of shape (b, d, l)28 y (torch.Tensor): tensor of shape (b, d, l)29 r@ � �����rB TrD )�sizer! r" r# �viewr r � transposer r r � unsqueezer �cumsum�sumr' r r$ �to�dtyper& ) r* rG �filters�argsr, �b�l�_rH rI rJ �ys r. �forward_referencez(ReBasedLinearAttention.forward_referenceZ s� � � �$�$�&���1�a��+�+�m�,�d�k�k�-�.H�$�+�+�Vc�Jd�a�1��
�F�F�1�a�� � ��)9�)9� :� D� D�Q�� J��
�F�F�1�a��1�1�4�3C�3C� D� N� N�q�RS� T��
�F�F�1�a��1�1�4�=�=� A� K� K�A�q� Q�� � � ��"�D�$4�$4�Q�$7�1���+�+�b�/�1�;�;�r�?�A�K�K��O�a�1�� �;�;��q�1�u�n�n�Q�'�'�,�,�R�0�Q����!��_�4I�4I�"�4M�PT�PX�PX�4X�Y�A��q�1�u�k�k�!�T�*�*�/�/��3��A�E�E�!�T�N�8J�7O�7O�PR�7S�VZ�V^�V^�7^�_�A� �a�/� 0�� �K�K����]� 0� 0�1� 2�� �L�L��O���t�t�M�'�'�(�(r/ ) i � r_ r_ TTTTg�h㈵��>r N)r �intr r` r r` r r` r r` r �Optional[bool]r ra r ra r �boolr' �floatr �strr+ z
Optional[int]�returnr )r9 z nn.Module)rG � torch.Tensor)N)rG rf rX rf )�__name__�30__module__� __qualname__r r) rL r^ �
__classcell__)r- s @r. r r s� �� � � �#%��$(�#'�$(��� �#'�(-��(-� �(-� � (-�31 !� (-� �
(-� "�(-� !�(-� "�(-� �(-� �(-� �(-� !�(-� 32 � (-�T)��$)� )r/ r �__main__� i Tr )r r )� retain_graphr
�Pass)'�__doc__�33__future__r �typingr �torch�torch.nnr �einopsr �fla.modules.feature_mapr �fla.ops.linear_attnr r �fla.ops.rebasedr �Moduler rg �batch�seq_lenr �float32rW �randnrV �cuda�requires_grad_r= �dy�modelr] �backward�grad�x_grad�printr �y2� r/ r. �<module>r� sA ��� #� � � � � 5� J� ,�_)�R�Y�Y� _)�D �z��
�E��G��K� �M�M�E��� � �E�7�K�0�3�3�E�:�?�?�A�P�P�QU�V�A� ����U�G�[� 1� 4� 4�U� ;� @� @� B�B� "�{�� L� O� O�PU� V� [� [� ]�E�
�a��A��J�J�r��J�%��V�V�T�N�F�A�F� �%�*�*�� �E�J� �q��B� �%�*�*���K�K��O� 34�&�M�' r/