declare-lab/tango2

1a

2��'d��@s>dd
lm
Z

ddlmZ
ddlmZmZ
Gdd�dej�ZdS)�)
�Optional)
�nn�
)�Transformer2DModel�Transformer2DModelOutputcsbeZ
dZd
Zdeeeeeeeeeeeeeeeeed3��f
dd�
Z	ded�
dd�Z4�ZS)�DualTransformer2DModelaw5    Dual transformer wrapper that combines two `Transformer2DModel`s for mixed inference.6 7    Parameters:8        num_attention_heads (`int`, *optional*, defaults to 16): The number of heads to use for multi-head attention.9        attention_head_dim (`int`, *optional*, defaults to 88): The number of channels in each head.10        in_channels (`int`, *optional*):11            Pass if the input is continuous. The number of channels in the input and output.12        num_layers (`int`, *optional*, defaults to 1): The number of layers of Transformer blocks to use.13        dropout (`float`, *optional*, defaults to 0.1): The dropout probability to use.14        cross_attention_dim (`int`, *optional*): The number of encoder_hidden_states dimensions to use.15        sample_size (`int`, *optional*): Pass if the input is discrete. The width of the latent images.16            Note that this is fixed at training time as it is used for learning a number of position embeddings. See17            `ImagePositionalEmbeddings`.18        num_vector_embeds (`int`, *optional*):19            Pass if the input is discrete. The number of classes of the vector embeddings of the latent pixels.20            Includes the class for the masked latent pixel.21        activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward.22        num_embeds_ada_norm ( `int`, *optional*): Pass if at least one of the norm_layers is `AdaLayerNorm`.23            The number of diffusion steps used during training. Note that this is fixed at training time as it is used24            to learn a number of embeddings that are added to the hidden states. During inference, you can denoise for25            up to but not more than steps than `num_embeds_ada_norm`.26        attention_bias (`bool`, *optional*):27            Configure if the TransformerBlocks' attention should contain a bias parameter.28    ��XNr�� F�geglu��num_attention_heads�attention_head_dim�in_channels�29num_layers�dropout�norm_num_groups�cross_attention_dim�attention_bias�sample_size�num_vector_embeds�
activation_fn�num_embeds_ada_normc

s\t��
�
t���
��������	�30�fd
d�td�
D�
�
|_d|_ddg|_ddg|_dS)	Nc
s,g|]$}
t����	����
��31��d��qS)
r
)
r)�.0�
_�rrrrrrrrrrrr��Y/home/deep/Projects/audio_diffusion/diffusers/src/diffusers/models/dual_transformer_2d.py�32<listcomp>As�











�z3DualTransformer2DModel.__init__.<locals>.<listcomp>�g�?�Mi

rr
)	�super�__init__r�33ModuleList�range�transformers�	mix_ratio�condition_lengths�transformer_index_for_condition)
�selfrrrrrrrrrrrr�
�	__class__rrr#0s34

 ��35zDualTransformer2DModel.__init__T)
�return_dictcCs�|
}g}d
}	td�
D]d}36|dd�|	|	|j
|37�f}|j|38}|j|||||dd�d
}
|�|
|�

|	|j
|397}	q|d
|j|dd|j}||}|s�|f
St|d�
S)a4041        Args:42            hidden_states ( When discrete, `torch.LongTensor` of shape `(batch size, num latent pixels)`.43                When continuous, `torch.FloatTensor` of shape `(batch size, channel, height, width)`): Input44                hidden_states45            encoder_hidden_states ( `torch.LongTensor` of shape `(batch size, encoder_hidden_states dim)`, *optional*):46                Conditional embeddings for cross attention layer. If not given, cross-attention defaults to47                self-attention.48            timestep ( `torch.long`, *optional*):49                Optional timestep to be applied as an embedding in AdaLayerNorm's. Used to indicate denoising step.50            attention_mask (`torch.FloatTensor`, *optional*):51                Optional attention mask to be applied in Attention52            return_dict (`bool`, *optional*, defaults to `True`):53                Whether or not to return a [`models.unet_2d_condition.UNet2DConditionOutput`] instead of a plain tuple.54 55        Returns:56            [`~models.transformer_2d.Transformer2DModelOutput`] or `tuple`:57            [`~models.transformer_2d.Transformer2DModelOutput`] if `return_dict` is True, otherwise a `tuple`. When58            returning a tuple, the first element is the sample tensor.59        r
r NF)�encoder_hidden_states�timestep�cross_attention_kwargsr-r)
�sample)r%r(r)r&�appendr'r)r*�
hidden_statesr.r/�attention_maskr0r-�input_statesZencoded_states�tokens_start�
iZcondition_stateZtransformer_indexZ
encoded_state�
output_statesrrr�forwardas,

60





��
 

zDualTransformer2DModel.forward)rr	Nrr61rNFNNrN)NNNT)�__name__�62__module__�__qualname__�__doc__�intr�float�bool�strr#r9�
__classcell__rrr+rrsD











�










�5


��rN)	�typingr�torchr�transformer_2drr�Modulerrrrr�<module>s