declare-lab/tango2

1a

2�0d]@�@s�dd
lm
Z

ddlmZmZmZ
ddlZddlm
m	Z3
ddlmZ
ddlmZm
Z

ddlmZ
ddlmZmZ
d	d4lmZ
d	dlmZ
d	dlmZ
e
Gd
d�de��
ZGdd�dee�ZdS)�)
�	dataclass)�Any�Dict�OptionalN)
�nn�)�ConfigMixin�register_to_config)
�ImagePositionalEmbeddings)�5BaseOutput�	deprecate�
�
�BasicTransformerBlock)
�6PatchEmbed)
�7ModelMixinc@seZ
dZUd
Zejed<dS)�Transformer2DModelOutputao
8    Args:9        sample (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)` or `(batch size, num_vector_embeds - 1, num_latent_pixels)` if [`Transformer2DModel`] is discrete):10            Hidden states conditioned on `encoder_hidden_states` input. If discrete, returns probability distributions11            for the unnoised latent pixels.12    �sampleN)�__name__�13__module__�__qualname__�__doc__�torch�FloatTensor�__annotations__�rr�T/home/deep/Projects/audio_diffusion/diffusers/src/diffusers/models/transformer_2d.pyrs14rcs�eZ
dZd
Zedeeeeeeeeeeeeeeeeeee	eeeeee	ed��f
d
d�
�
Z15dejeejeej
eej
ee	efeejeejed�dd�Z�ZS)�Transformer2DModela11617    Transformer model for image-like data. Takes either discrete (classes of vector embeddings) or continuous (actual18    embeddings) inputs.19 20    When input is continuous: First, project the input (aka embedding) and reshape to b, t, d. Then apply standard21    transformer action. Finally, reshape to image.22 23    When input is discrete: First, input (classes of latent pixels) is converted to embeddings and has positional24    embeddings applied, see `ImagePositionalEmbeddings`. Then apply standard transformer action. Finally, predict25    classes of unnoised image.26 27    Note that it is assumed one of the input classes is the masked latent pixel. The predicted classes of the unnoised28    image do not contain a prediction for the masked pixel as the unnoised image cannot be masked.29 30    Parameters:31        num_attention_heads (`int`, *optional*, defaults to 16): The number of heads to use for multi-head attention.32        attention_head_dim (`int`, *optional*, defaults to 88): The number of channels in each head.33        in_channels (`int`, *optional*):34            Pass if the input is continuous. The number of channels in the input and output.35        num_layers (`int`, *optional*, defaults to 1): The number of layers of Transformer blocks to use.36        dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use.37        cross_attention_dim (`int`, *optional*): The number of encoder_hidden_states dimensions to use.38        sample_size (`int`, *optional*): Pass if the input is discrete. The width of the latent images.39            Note that this is fixed at training time as it is used for learning a number of position embeddings. See40            `ImagePositionalEmbeddings`.41        num_vector_embeds (`int`, *optional*):42            Pass if the input is discrete. The number of classes of the vector embeddings of the latent pixels.43            Includes the class for the masked latent pixel.44        activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward.45        num_embeds_ada_norm ( `int`, *optional*): Pass if at least one of the norm_layers is `AdaLayerNorm`.46            The number of diffusion steps used during training. Note that this is fixed at training time as it is used47            to learn a number of embeddings that are added to the hidden states. During inference, you can denoise for48            up to but not more than steps than `num_embeds_ada_norm`.49        attention_bias (`bool`, *optional*):50            Configure if the TransformerBlocks' attention should contain a bias parameter.51    ��XNr
�� F�geglu�52layer_normT)�num_attention_heads�attention_head_dim�in_channels�out_channels�53num_layers�dropout�norm_num_groups�cross_attention_dim�attention_bias�sample_size�num_vector_embeds�54patch_size�
activation_fn�num_embeds_ada_norm�use_linear_projection�only_cross_attention�upcast_attention�	norm_type�norm_elementwise_affinecs�t��
�
||_�|_�|_���|du
o2|du|_|du
|_|du
oN|du
|_�d
kr��	du
r�d|j�d�}t	dd|dd�
d�|jr�|jr�t55d	|�d56|�d��
�
nR|jr�|jr�t57d|�d
|�d��
�
n.|js�|js�|js�t58d|�d|�d|�d��
�
|j�
rL||_tj
j||ddd�|_|�
r4t
�|��|_nt
j|�dddd�|_n�|j�
r�|59du
�
sfJd�
�
|du
�
sxJd�
�
|60|_|61|_||_|j|j|_t|�|j|jd�|_n@|j�
r�|62du
�
s�Jd�
�
|63|_|64|_||_t|65|66||�d�|_t
���
��������	�67�fdd�t|�
D�
�
|_|du�r2|n||_|j�rn|�rVt
��|�|_ nt
j�|dddd�|_ nn|j�r�t
�!��
|_"t
��|jd�|_#nD|j�r�t
j!�ddd �|_"t
��d!��|_$t
��|||j�|_%dS)"Nr#z&The configuration file of this model: a�
 is outdated. `norm_type` is either not set or incorrectly set to `'layer_norm'`.Make sure to set `norm_type` to `'ada_norm'` in the config. Please make sure to update the config accordingly as leaving `norm_type` might led to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the `transformer/config.json` fileznorm_type!=num_embeds_ada_normz1.0.0F)
�
standard_warn�ada_normz"Cannot define both `in_channels`: z and `num_vector_embeds`: zE. Make sure that either `in_channels` or `num_vector_embeds` is None.z(Cannot define both `num_vector_embeds`: z and `patch_size`: zE. Make sure that either `num_vector_embeds` or `num_patches` is None.zHas to define `in_channels`: z, `num_vector_embeds`: z, or patch_size: zQ. Make sure that `in_channels`, `num_vector_embeds` or `num_patches` is not None.g�����ư>T)�68num_groups�num_channels�eps�affiner
r
)�kernel_size�stride�paddingz?Transformer2DModel over discrete input must provide sample_sizez=Transformer2DModel over discrete input must provide num_embed)�	num_embed�	embed_dim�height�widthz>Transformer2DModel over patched input must provide sample_size)rBrCr/r&rAc
s,g|]$}
t�������	�
�69���d��qS)
)	r)r+r0r1r,r3r4r5r6r)�.0�
d�r0r,r%r+r)�	inner_dimr6r5r$r1r3r4rr�70<listcomp>�s�











�z/Transformer2DModel.__init__.<locals>.<listcomp>)�elementwise_affiner;r)&�super�__init__r2r$r%�is_input_continuous�is_input_vectorized�is_input_patches�	__class__r�71ValueErrorr&rr�	GroupNorm�norm�Linear�proj_in�Conv2drBrCr.Znum_latent_pixelsr72�latent_image_embeddingr/r�	pos_embed�73ModuleList�range�transformer_blocksr'�proj_out�	LayerNorm�norm_out�out�74proj_out_1�75proj_out_2)�selfr$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6�deprecation_message�
rOrFrrKOs�76




77
�


�

�


��









�78







�	
 ��








zTransformer2DModel.__init__)�
hidden_states�encoder_hidden_states�timestep�class_labels�cross_attention_kwargs�attention_mask�encoder_attention_mask�return_dictc	79Csx|jr�|
j
\}	}80}}|
}
|�|
�
}
|js\|�|
�
}
|
j
d
}|
�dddd
��|	|||�}
q�|
j
d
}|
�dddd
��|	|||�}
|�|
�
}
n"|jr�|�|
�
}
n|j	r�|�81|
�
}
|jD]}||
||||||d�}
q�|j�
rF|j�
s|
�|	|||��ddd
d���}
|�
|
�
}
n*|�
|
�
}
|
�|	|||��ddd
d���}
|
|
}�
n|j�
r�|�|
�
}
|�|
�
}|�ddd
�}tj|��d
d���}n�|j	�rb|jdjj|||
jd�}|�t�|�
�
jdd
d�\}}|�|
�
d
|dd�df|dd�df}
|�|
�
}
t|
j
d
d	�
}}|
jd82|||j|j|jfd�
}
t�d|
�}
|
jd83|j||j||jfd�
}|�sn|f
St |d
�
S)a�84        Args:85            hidden_states ( When discrete, `torch.LongTensor` of shape `(batch size, num latent pixels)`.86                When continuous, `torch.FloatTensor` of shape `(batch size, channel, height, width)`): Input87                hidden_states88            encoder_hidden_states ( `torch.LongTensor` of shape `(batch size, encoder_hidden_states dim)`, *optional*):89                Conditional embeddings for cross attention layer. If not given, cross-attention defaults to90                self-attention.91            timestep ( `torch.LongTensor`, *optional*):92                Optional timestep to be applied as an embedding in AdaLayerNorm's. Used to indicate denoising step.93            class_labels ( `torch.LongTensor` of shape `(batch size, num classes)`, *optional*):94                Optional class labels to be applied as an embedding in AdaLayerZeroNorm. Used to indicate class labels95                conditioning.96            attention_mask ( `torch.Tensor` of shape (batch size, num latent pixels), *optional* ).97                Bias to add to attention scores.98            encoder_attention_mask ( `torch.Tensor` of shape (batch size, num encoder tokens), *optional* ).99                Bias to add to cross-attention scores.100            return_dict (`bool`, *optional*, defaults to `True`):101                Whether or not to return a [`models.unet_2d_condition.UNet2DConditionOutput`] instead of a plain tuple.102 103        Returns:104            [`~models.transformer_2d.Transformer2DModelOutput`] or `tuple`:105            [`~models.transformer_2d.Transformer2DModelOutput`] if `return_dict` is True, otherwise a `tuple`. When106            returning a tuple, the first element is the sample tensor.107        r
r
r�)rirerjrfrhrg)
�dim)
�hidden_dtypeNg�?�����)
�shapeznhwpqc->nchpwq)
r)!rLrprRr2rT�permute�reshaperMrVrNrWrZ�108contiguousr[r]r^�
F�log_softmax�double�float�norm1�emb�dtyper_�silu�chunkr`�intr/r'r�einsumr)rardrerfrgrhrirjrk�batch�
_rBrC�residualrG�block�output�logits�conditioning�shift�scalerrr�forward�sn%

109

110
111
 112





113114







�

 
115
 

116
117

�
.
118

�

�
zTransformer2DModel.forward)rrNNr
r r!NFNNNr"NFFFr#T)NNNNNNT)rrrrr	r}rrw�bool�strrKr�Tensor�119LongTensorrrr��
__classcell__rrrcrr)sx
%

















�

















�120





�



121


�r)�dataclassesr�typingrrrrZtorch.nn.functionalr�122functionalrt�configuration_utilsrr	Zmodels.embeddingsr123�utilsrr�	attentionr�124embeddingsr�modeling_utilsrrrrrrr�<module>s