Apppublic
declare-lab/tango2

sourceHugging Faceupdated 2y agoView on Hugging Face
92likes
rootdiffuserssrcdiffusersmodels__pycache__attention.cpython-39.pyc
attention.cpython-39.pyc170 linesDownload Raw Back to __pycache__
1a

2�0d�O�@s,ddlZddlmZmZmZmZddlZddlmm	Z3ddlmZddlmZddl
mZddlmZe�r�ddlZddlZndZGd	d4�d5ej�ZGdd�dej�ZGd
d�dej�ZGdd�dej�ZGdd�dej�ZGdd�dej�ZGdd�dej�ZGdd�dej�ZGdd�dej�ZdS)�N)�Any�Callable�Dict�Optional)�nn�)�is_xformers_available�)�	Attention)�CombinedTimestepLabelEmbeddingscsbeZdZdZdeeeeeed��fdd�
Zd	d6�Zdd�Z	de7eed
�dd�Zdd�Z
�ZS)�AttentionBlockaU8    An attention block that allows spatial positions to attend to each other. Originally ported from here, but adapted9    to the N-d case.10    https://github.com/hojonathanho/diffusion/blob/1e0dceb3b3495bbe19116a5e1b3596cd0706c543/diffusion_tf/models/unet.py#L66.11    Uses three q, k, v linear layers to compute attention.12 13    Parameters:14        channels (`int`): The number of channels in the input and output.15        num_head_channels (`int`, *optional*):16            The number of channels in each head. If None, then `num_heads` = 1.17        norm_num_groups (`int`, *optional*, defaults to 32): The number of groups to use for group norm.18        rescale_output_factor (`float`, *optional*, defaults to 1.0): The factor to rescale the output by.19        eps (`float`, *optional*, defaults to 1e-5): The epsilon value to use for group norm.20    N� ��?��h㈵��>)�channels�num_head_channels�norm_num_groups�rescale_output_factor�epscs�t���||_|dur ||nd|_||_tj|||dd�|_t�||�|_	t�||�|_21t�||�|_||_tj||dd�|_
d|_d|_dS)Nr	T)�num_channels�22num_groupsr�affine��biasF)�super�__init__r�	num_headsZ
num_head_sizer�	GroupNorm�23group_norm�Linear�query�key�valuer�	proj_attn�(_use_memory_efficient_attention_xformers�
_attention_op)�selfrrrrr��	__class__��O/home/deep/Projects/audio_diffusion/diffusers/src/diffusers/models/attention.pyr3s24zAttentionBlock.__init__cCsL|j\}}}|j}|�|||||�}|�dddd��|||||�}|S�Nrrr	���shaper�reshape�permute�r&�tensor�25batch_size�seq_len�dim�	head_sizer)r)r*�reshape_heads_to_batch_dimMs26"z)AttentionBlock.reshape_heads_to_batch_dimcCsL|j\}}}|j}|�|||||�}|�dddd��|||||�}|Sr+r-r1r)r)r*�reshape_batch_dim_to_headsTs27"z)AttentionBlock.reshape_batch_dim_to_heads)�'use_memory_efficient_attention_xformers�attention_opc28Cs�|r�t�stddd��nptj��s,td��n\z2tj�tj	ddd�tj	ddd�tj	ddd��}Wn(t29y�}z|�WYd}~n30d}~00||_||_dS)NzeRefer to https://github.com/facebookresearch/xformers for more information on how to install xformers�xformers)�namezvtorch.cuda.is_available() should be True but is False. xformers' memory efficient attention is only available for GPU )r	r�(�cuda)�device)
r�ModuleNotFoundError�torchr>�is_available�31ValueErrorr;�ops�memory_efficient_attention�randn�	Exceptionr$r%)r&r9r:�_�er)r)r*�+set_use_memory_efficient_attention_xformers[s(�32��z:AttentionBlock.set_use_memory_efficient_attention_xformersc
CsZ|}|j\}}}}|�|�}|�||||��dd�}|�|�}|�|�}|�|�}	dt�|j	|j33�}34|�|�}|�|�}|�|	�}	|jr�t
jj|||	d|jd�}|�|j�}nhtjtj|jd|jd|jd|j|jd�||�dd�d|35d�}tj|��dd	��|j�}t�||	�}|�|�}|�|�}|�dd��||||�}|||j}|S)36Nr	r)Z	attn_bias�opr)�dtyper?����������)�beta�alpha�r5)r.r�view�	transposer r!r"�math�sqrtrrr7r$r;rDrEr%�torLrA�baddbmm�emptyr?�softmax�float�type�bmmr8r#r/r)
r&�
hidden_statesZresidual�batch�channel�height�widthZ37query_projZkey_projZ38value_proj�scaleZattention_scoresZattention_probsr)r)r*�forwardysH39404142434445��46�
4748zAttentionBlock.forward)Nr
rr)N)�__name__�49__module__�__qualname__�__doc__�intrrZrr7r8�boolrrJrc�
__classcell__r)r)r'r*r!s&����rcs�eZdZdZdeeeeeeeeeeeeeeed�
�fd	d50�
Zde	j51ee	j52ee	j53ee	j54ee	jeee
fee	jd�dd
�Z�ZS)�BasicTransformerBlocka�55    A basic Transformer block.56 57    Parameters:58        dim (`int`): The number of channels in the input and output.59        num_attention_heads (`int`): The number of heads to use for multi-head attention.60        attention_head_dim (`int`): The number of channels in each head.61        dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use.62        cross_attention_dim (`int`, *optional*): The size of the encoder_hidden_states vector for cross attention.63        only_cross_attention (`bool`, *optional*):64            Whether to use only cross-attention layers. In this case two cross attention layers are used.65        double_self_attention (`bool`, *optional*):66            Whether to use two self-attention layers. In this case no cross attention layers are used.67        activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward.68        num_embeds_ada_norm (:69            obj: `int`, *optional*): The number of diffusion steps used during training. See `Transformer2DModel`.70        attention_bias (:71            obj: `bool`, *optional*, defaults to `False`): Configure if the attentions should contain a bias parameter.72    �N�gegluFT�73layer_norm)
r5�num_attention_heads�attention_head_dim�cross_attention_dim�
activation_fn�num_embeds_ada_norm�attention_bias�only_cross_attention�double_self_attention�upcast_attention�norm_elementwise_affine�	norm_type�
final_dropoutc	sFt���|	|_|duo|
dk|_|duo0|
dk|_|
dvrZ|durZtd|
�d|
�d���t||||||	rn|nd|d�|_t||||d�|_	|dus�|74r�t||75s�|nd|||||d	�|_76nd|_77|jr�t||�|_n$|jr�t
||�|_ntj||d78�|_|du�s|79�r,|j�rt||�ntj||d80�|_nd|_tj||d81�|_dS)N�
ada_norm_zero�ada_norm)r|r{z`norm_type` is set to zw, but `num_embeds_ada_norm` is not defined. Please make sure to define `num_embeds_ada_norm` if setting `norm_type` to �.)�	query_dim�heads�dim_head�dropoutrrqrw)r�rrrz)r~rqrr�r�rrw��elementwise_affine)rrru�use_ada_layer_norm_zero�use_ada_layer_normrCr82�attn1�FeedForward�ff�attn2�AdaLayerNorm�norm1�AdaLayerNormZeror�	LayerNorm�norm2�norm3)r&r5rorpr�rqrrrsrtrurvrwrxryrzr'r)r*r�sV83��84�8586�87 88��zBasicTransformerBlock.__init__)r]�attention_mask�encoder_hidden_states�encoder_attention_mask�timestep�cross_attention_kwargs�class_labelscCs>|jr|�||�}n0|jr:|j||||jd�\}}	}89}}n90|�|�}|durP|ni}|j|f|jrf|nd|d�|��}
|jr�|	�d�|
}
|
|}|jdur�|jr�|�||�n|�|�}|j|f||d�|��}
|
|}|�	|�}|j�r|d|dd�df|91dd�df}|�92|�}|j�r2|�d�|}||}|S)N��hidden_dtype)r�r�r	)r�r�r�rLr�ru�	unsqueezer�r�r�r�)r&r]r�r�r�r�r�r��norm_hidden_states�gate_msa�	shift_mlp�	scale_mlp�gate_mlp�attn_outputZ	ff_outputr)r)r*rcsN9394�95���96����97(98zBasicTransformerBlock.forward)rlNrmNFFFFTrnF)NNNNNN)rdrerfrgrhr�strrirrA�FloatTensor�99LongTensorrrrcrjr)r)r'r*rk�sT��Q�100�rkcs>eZdZdZdeeeeeeed��fdd	�
Z	d101d�Z102�ZS)
r�aV103    A feed-forward layer.104 105    Parameters:106        dim (`int`): The number of channels in the input.107        dim_out (`int`, *optional*): The number of channels in the output. If not given, defaults to `dim`.108        mult (`int`, *optional*, defaults to 4): The multiplier to use for the hidden dimension.109        dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use.110        activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward.111        final_dropout (`bool` *optional*, defaults to False): Apply a final dropout.112    N�rlrmF)r5�dim_out�multr�rrrzc	s�t���t||�}|dur"|n|}|dkr8t||�}|dkrPt||dd�}n&|dkrdt||�}n|dkrvt||�}t�g�|_|j�	|�|j�	t�113|��|j�	t�||��|r�|j�	t�114|��dS)N�geluzgelu-approximate�tanh��approximatermzgeglu-approximate)rrrh�GELU�GEGLU�ApproximateGELUr�115ModuleList�net�append�Dropoutr)	r&r5r�r�r�rrrzZ	inner_dim�act_fnr'r)r*r^s"	116117118zFeedForward.__init__cCs|jD]}||�}q|S�N)r�)r&r]�moduler)r)r*rcs119120zFeedForward.forward)Nr�rlrmF)rdrerfrgrhrrZr�rirrcrjr)r)r'r*r�Qs��!r�cs<eZdZdZd121eeed��fdd�
Zdd�Zdd	�Z�Z	S)r�z]122    GELU activation function with tanh approximation support with `approximate="tanh"`.123    �none)�dim_inr�r�cs"t���t�||�|_||_dSr�)rrrr�projr�)r&r�r�r�r'r)r*r�s124z
GELU.__init__cCs@|jjdkrtj||jd�Stj|jtjd�|jd�j|jd�S)N�mpsr��rL)	r?r[�Fr�r�rVrA�float32rL�r&�gater)r)r*r��sz	GELU.gelucCs|�|�}|�|�}|Sr�)r�r�)r&r]r)r)r*rc�s125126zGELU.forward)r�)127rdrerfrgrhr�rr�rcrjr)r)r'r*r��sr�cs8eZdZdZeed��fdd�Zdd�Zdd�Z�ZS)	r�z�128    A variant of the gated linear unit activation function from https://arxiv.org/abs/2002.05202.129 130    Parameters:131        dim_in (`int`): The number of channels in the input.132        dim_out (`int`): The number of channels in the output.133    �r�r�cs t���t�||d�|_dS)Nr�rrrrr��r&r�r�r'r)r*r�s134zGEGLU.__init__cCs4|jjdkrt�|�St�|jtjd��j|jd�S)Nr�r�)r?r[r�r�rVrAr�rLr�r)r)r*r��s135z136GEGLU.gelucCs&|�|�jddd�\}}||�|�S)NrrMrQ)r��chunkr�)r&r]r�r)r)r*rc�sz
GEGLU.forward)	rdrerfrgrhrr�rcrjr)r)r'r*r��sr�cs0eZdZdZeed��fdd�Zdd�Z�ZS)r�z�137    The approximate form of Gaussian Error Linear Unit (GELU)138 139    For more details, see section 2: https://arxiv.org/abs/1606.08415140    r�cst���t�||�|_dSr�r�r�r'r)r*r�s141zApproximateGELU.__init__cCs|�|�}|t�d|�S)Ng�Zd;�?)r�rA�sigmoid)r&�xr)r)r*rc�s142zApproximateGELU.forward)rdrerfrgrhrrcrjr)r)r'r*r��sr�cs(eZdZdZ�fdd�Zdd�Z�ZS)r�zA143    Norm layer modified to incorporate timestep embeddings.144    csHt���t�||�|_t��|_t�||d�|_tj	|dd�|_145dS)NrFr�)rrr�	Embedding�emb�SiLU�silur�linearr��norm�r&�
embedding_dim�num_embeddingsr'r)r*r�s146147148zAdaLayerNorm.__init__cCs@|�|�|�|���}t�|d�\}}|�|�d||}|S)Nrr	)r�r�r�rAr�r�)r&r�r�r�rb�shiftr)r)r*rc�szAdaLayerNorm.forward�rdrerfrgrrcrjr)r)r'r*r��sr�cs*eZdZdZ�fdd�Zddd�Z�ZS)r�z;149    Norm layer adaptive layer norm zero (adaLN-Zero).150    csLt���t||�|_t��|_tj|d|dd�|_tj	|ddd�|_151dS)N�TrFg�����ư>)r�r)rrrr�rr�r�rr�r�r�r�r'r)r*r�s152153154zAdaLayerNormZero.__init__Nc	Csr|�|�|j|||d���}|jddd�\}}}}	}155}|�|�d|dd�df|dd�df}|||	|156|fS)Nr�r�r	rQ)r�r�r�r�r�)r&r�r�r�r�r�Z	shift_msaZ	scale_msar�r�r�r�r)r)r*rc�s.zAdaLayerNormZero.forward)Nr�r)r)r'r*r��s	r�cs<eZdZdZd	eeeeeed��fdd�
Zdd�Z	�Z157S)158�AdaGroupNormzF159    GroupNorm layer modified to incorporate timestep embeddings.160    Nr)r��out_dimrr�rcs�t���||_||_d|_|dkr0dd�|_n:|dkrDt��|_n&|dkrXt��|_n|dkrjt��|_t�	||d�|_161dS)N�swishcSs162t�|�Sr�)r�r�)r�r)r)r*�<lambda>��z'AdaGroupNorm.__init__.<locals>.<lambda>�mishr�r�r)rrrr�actr�Mishr�r�rr�)r&r�r�rr�rr'r)r*r�s163164zAdaGroupNorm.__init__cCsl|jr|�|�}|�|�}|dd�dd�ddf}|jddd�\}}tj||j|jd�}|d||}|S)Nrr	rQ)r)r�r�r�r�rrr)r&r�r�rbr�r)r)r*rcs165166zAdaGroupNorm.forward)Nr)rdrerfrgrhrr�rZrrcrjr)r)r'r*r��s��r�)rT�typingrrrrrAZtorch.nn.functionalr�167functionalr�Zutils.import_utilsr�attention_processorr168�169embeddingsrr;Zxformers.ops�Modulerrkr�r�r�r�r�r�r�r)r)r)r*�<module>s,170!4