msj19/gated_deltaproduct
05
1U
2 *�gX � @ sb d Z ddlZddlmZ ddlmZ ddlmZ ddlm Z ddl3m Z m Z G dd� dej
�ZdS ) zf4Linear attention in Based.5https://github.com/HazyResearch/zoology/blob/main/zoology/mixers/based.py6� N)� rearrange)�TaylorFeatureMap)�parallel_based)�chunk_linear_attn�fused_chunk_linear_attnc 7 sh e Zd Zdeeeeeeeed�� fdd �
Zej d8�d d �Z9e j d
�dd�Z
de j e j d�dd�Z� ZS )�BasedLinearAttention� � �10taylor_exp��-���q=T�parallel)� hidden_size� feature_dim�num_key_value_heads� num_heads� feature_name�eps�causal�modec s� t � �� || _|| _|| _|| _|| _|| _| j| j | _|| _ t11j | j| j| j dd�| _ t12j | j| j| j dd�| _
t13j | j| j| j dd�| _t14j | j| j | jdd�| _t15�� | _t|�| _|| _| �| j� d S )NF)�bias)�super�__init__r
r r r r r �head_dimr �nn�Linear�q_proj�k_proj�v_proj�o_proj�Identity�dropoutr � feature_mapr �apply�_initialize_weights) �selfr
r r r r r r r �� __class__� �0/public/liguoqi/msj/gen_test/fla/layers/based.pyr s" 16 1718z BasedLinearAttention.__init__)�modulec C sP t |dd�rd S t|tj�rFtjj|jdd� |jd k rFtj�|j� d|_ d S )N�_is_hf_initializedFg�;f���?)�gainT)19�getattr�20isinstancer r �init�xavier_uniform_�weightr �zeros_r* )r$ r) r' r' r( r# 3 s 21z(BasedLinearAttention._initialize_weights)�
hidden_statesc s� � j }� �|�� �|�� �|� }}}t� fdd�|||g�\}}}|dkrx� �|�� �|� }}t|||ddd�}n\|dkr�� �|�� �|� }}t|||ddd�}n*|dkr�|jd d22ks�t �t23|||dd�}t |d �}� � |�}� �
|�}|S ) Nc s t | d� jd�S )Nzb l (h d) -> b h l d)�h)r r )�x�r$ r' r( �<lambda>? � z.BasedLinearAttention.forward.<locals>.<lambda>� fused_chunkT� )� normalize�scale�chunkr ������ �b h l d -> b l (h d))r r r r �mapr! r r �shape�AssertionErrorr r r r )r$ r2 �kwargsr �q�k�v�or' r5 r( �forward< s " 242526z BasedLinearAttention.forwardN)r2 �filtersc O s^ |� � \}}}| �|�| �|�| �|� }} }27|�||| j| j��dd�}| �||| j| j��dd�} |28�||| j| j ��dd�}29| �30|�| �31| � }} |� d�| � d�|32� d� }} }33| j r�|| |34 �
d� �d�|| �
d� �d�| j } n6|| |35 �dd� �d�|| �dd� �d�| j } t| d�} | �| �|j��} | �| �} | �|j�S )zq36 x (torch.Tensor): tensor of shape (b, d, l)37 y (torch.Tensor): tensor of shape (b, d, l)38 r9 � �����r= Tr? )�sizer r r �viewr r � transposer r r! � unsqueezer �cumsum�sumr r r �to�dtyper ) r$ r2 rI �argsrC �b�l�_rD rE rF �yr' r' r( �forward_referenceP s " "463940z&BasedLinearAttention.forward_reference)r r r r41 r Tr )N)�__name__�42__module__� __qualname__�int�str�float�boolr r �Moduler# �torch�TensorrH rY �
__classcell__r' r' r% r( r s( �� r )�__doc__rb �torch.nnr �einopsr Zfla.modules.feature_mapr Z
fla.ops.basedr Zfla.ops.linear_attnr r ra r r' r' r' r( �<module> s