declare-lab/tango2
92
1a
2 �0d�� � @ s� d dl mZmZmZmZ d dlZd dlZd dlmZ ddl m3Z4m Z ddl m
Z
mZ ddlmZ ddlmZmZmZmZmZmZmZ dd lmZmZ dDd d
�Z dEdd�Z G dd� dej �Z G dd� dej �Z G dd� dej �Z!G dd� dej �Z"G dd� dej �Z#G dd� dej �Z$G d d � d ej �Z%G d d � d ej �Z&G d d!� d!ej �Z'G d"d#� d#ej �Z(G d$d%� d%ej �Z)G d&d'� d'ej �Z*G d(d)� d)ej �Z+G d*d+� d+ej �Z,G d,d-� d-ej �Z-G d.d/� d/ej �Z.G d0d1� d1ej �Z/G d2d3� d3ej �Z0G d4d5� d5ej �Z1G d6d7� d7ej �Z2G d8d9� d9ej �Z3G d:d;� d;ej �Z4G d<d=� d=ej �Z5G d>d?� d?ej �Z6G d@dA� dAej �Z7G dBdC� dCej �Z8dS )F� )�Any�Dict�Optional�TupleN)�nn� )� AdaGroupNorm�AttentionBlock)� Attention�AttnAddedKVProcessor)�DualTransformer2DModel)� Downsample2D�FirDownsample2D�
FirUpsample2D�
KDownsample2D� KUpsample2D�
ResnetBlock2D�5Upsample2D)�Transformer2DModel�Transformer2DModelOutputF�defaultc C s� | � d�r| dd � n| } | dkr>t|||||||| | |d�6S | dkr`t|||||||| |d� S | dkr�t|||||||| | ||d� S | d kr�|7d u r�td8��t|||||||| | |9|| |
|||d �S | d k� r�|10d u r�td
��t|||||||| |11||d� S | dk�r"t|||||||| |d� S | dk�rHt|||||||| ||d�12S | dk�rlt ||||||| | |d� S | dk�r�t13||||||| | ||d�14S | dk�r�t |||||||d�S | dk�r�t ||||||||15||�s�dndd �16S t| � d ���d S ) N�UNetRes� � DownBlock2D)17�18num_layers� in_channels� out_channels�
temb_channels�add_downsample�19resnet_eps�
resnet_act_fn�
resnet_groups�downsample_padding�resnet_time_scale_shift�ResnetDownsampleBlock2D) r r r r r r r r! r# �AttnDownBlock2D) r r r r r r r r! r" �attn_num_head_channelsr# �CrossAttnDownBlock2Dz>cross_attention_dim must be specified for CrossAttnDownBlock2D)r r r r r r r r! r" �cross_attention_dimr& �dual_cross_attention�use_linear_projection�only_cross_attention�upcast_attentionr# �SimpleCrossAttnDownBlock2DzDcross_attention_dim must be specified for SimpleCrossAttnDownBlock2D) r r r r r r r r! r( r& r# �SkipDownBlock2D) r r r r r r r r" r# �AttnSkipDownBlock2D)20r r r r r r r r" r& r# �DownEncoderBlock2D) r r r r r r r! r" r# �AttnDownEncoderBlock2D)21r r r r r r r! r" r& r# � KDownBlock2D)r r r r r r r �KCrossAttnDownBlock2DTF)22r r r r r r r r( r& �add_self_attention� does not exist.)
�23startswithr r$ r% �24ValueErrorr' r- r. r/ r0 r1 r2 r3 )�down_block_typer r r r r r r r& r! r( r" r) r* r+ r, r# � r9 �T/home/deep/Projects/audio_diffusion/diffusers/src/diffusers/models/unet_2d_blocks.py�get_down_block s. � � �
�25�
26� 27� 28� 29� 30� 31 � r; c C s� | � d�r| dd � n| } | dkr>t|||||||||32|d�33S | dkrbt|||||||||34|d�35S | dkr�| d u rztd��t|||||||||36| | | |
|||d�S | d kr�| d u r�td37��t|||||||||38| | |d � S | d k�rt|||||||||39| |d
� S | dk�r&t|||||||||d� S | dk�rLt||||||||| |d�40S | dk�rnt |||||||41|d�S | dk�r�t42|||||||43| |d� S | dk�r�t |||||||d�S | dk�r�t |||||||| | d� S t| � d���d S )Nr r � UpBlock2D)44r r r �prev_output_channelr � add_upsampler r r! r# �ResnetUpsampleBlock2D�CrossAttnUpBlock2Dz<cross_attention_dim must be specified for CrossAttnUpBlock2D)r r r r= r r> r r r! r( r& r) r* r+ r, r# �SimpleCrossAttnUpBlock2DzBcross_attention_dim must be specified for SimpleCrossAttnUpBlock2D) r r r r= r r> r r r! r( r& r# �
AttnUpBlock2D) r r r r= r r> r r r! r& r# �
SkipUpBlock2D) r r r r= r r> r r r# �AttnSkipUpBlock2D)45r r r r= r r> r r r& r# �UpDecoderBlock2D)r r r r> r r r! r# �AttnUpDecoderBlock2D) r r r r> r r r! r& r# �46KUpBlock2D)r r r r r> r r �KCrossAttnUpBlock2D) r r r r r> r r r( r& r5 )
r6 r< r? r7 r@ rA rB rC rD rE rF rG rH )�
up_block_typer r r r= r r> r r r& r! r( r) r* r+ r, r# r9 r9 r: � get_up_block� s, � � ��47�
48� 49� 50�51 52� 53� 54� rJ c s@ e Zd Zdeeeeeeeeeed �55� fd56d �
Zdd
d�Z� Z S )�UNetMidBlock2D� r �����ư>r �swish� T� �?)57r r �dropoutr r r# r r! �resnet_pre_norm�
add_attentionc
s� t � �� |d ur|n t|d d�}|58| _t||||||||| | d�59g}
g }t|�D ]P}| jrx|�t|| | ||d�� n60|�d � |
�t||||||||| | d�61� qTt� |�| _62t� |
�| _ d S )N� rO �63r r r �eps�groupsrQ �time_embedding_norm�
non_linearity�output_scale_factor�pre_norm��num_head_channels�rescale_output_factorrV �norm_num_groups) �super�__init__�minrS r �range�appendr r �64ModuleList�65attentions�resnets)�selfr r rQ r r r# r r! rR rS r& rZ rg rf �_�� __class__r9 r: ra t sZ 66�� ��67 68�� zUNetMidBlock2D.__init__Nc C sN | j d ||�}t| j| j dd � �D ]"\}}|d ur>||�}|||�}q&|S )Nr r �rg �ziprf )rh �
hidden_states�temb�attn�resnetr9 r9 r: �forward� s zUNetMidBlock2D.forward)69rL r rM r rN rO TTr rP )N�70�__name__�71__module__� __qualname__�int�float�str�boolra rr �
__classcell__r9 r9 rj r: rK s s. ��DrK c
sx e Zd Zdeeeeeeeeed � � fd d
�
Zdej e72ej e73ej e74ej e75e ee f e76ej ej d�dd�Z
� ZS )�UNetMidBlock2DCrossAttnrL r rM r rN rO TrP � F� r r rQ r r r# r r! rR c s� t � �� d| _|77| _|d ur"|n t|d d�}t||||||||| | d�78g}g }t|�D ]l}|
s�|�t|79||80 |d| |||d�� n |�t |81||82 |d| |d�� |�t||||||||| | d�83� qZt84� |�| _ t85� |�| _
d S )NTrT rO rU r )r r r( r_ r* r, �r r r( r_ )r` ra �has_cross_attentionr& rb r rc rd r r r re rf rg )rh r r rQ r r r# r r! rR r& rZ r( r) r* r, rg rf ri rj r9 r: ra � st 86�� ��
��87�� z UNetMidBlock2DCrossAttn.__init__N)rn ro �encoder_hidden_states�attention_mask�cross_attention_kwargs�encoder_attention_mask�returnc 88 C sV | j d ||�}t| j| j dd � �D ]*\}}||||||d�} | j}|||�}q&|S )Nr r �r� r� r� r� )rg rm rf �sample)89rh rn ro r� r� r� r� rp rq �outputr9 r9 r: rr s � z UNetMidBlock2DCrossAttn.forward)
rL r rM r rN rO Tr rP r} FFF)NNNNN�rt ru rv rw rx ry rz ra �torch� FloatTensorr r r rr r{ r9 r9 rj r: r| � sL ��W ��r| c s> e Zd Zdeeeeeeeeed90� � fd d �
Zddd�Z� Z S )� UNetMidBlock2DSimpleCrossAttnrL r rM r rN rO TrP r} r~ c
s� t � �� d| _|91| _|d ur"|n t|d d�}|| j | _t||||||||| | d�92g}
g }t|�D ]J}|�t ||| j|93| |ddt94� d� � |
�t||||||||| | d�95� qft � |�| _
t � |
�| _d S )NTrT rO rU � � query_dimr( �heads�dim_headZadded_kv_proj_dimr_ �biasZupcast_softmax� processor)r` ra r� r& rb � num_headsr rc rd r96 r r re rf rg )rh r r rQ r r r# r r! rR r& rZ r( rg rf ri rj r9 r: ra 0 sb 97 �� ��
�� z&UNetMidBlock2DSimpleCrossAttn.__init__Nc C sd |d ur |ni }| j d ||�}t| j| j dd � �D ](\}}||f||d�|��}|||�}q6|S )Nr r �r� r� rl )rh rn ro r� r� r� rp rq r9 r9 r: rr y s ��� z%UNetMidBlock2DSimpleCrossAttn.forward)98rL r rM r rN rO Tr rP r} )NNNNrs r9 r9 rj r: r� / s. ��J �r� c s@ e Zd Zdeeeeeeeeeed �99� fd100d �
Zdd
d�Z� Z S )r% rL r rM r rN rO TrP �101r r r rQ r r r# r r! rR c s� t � �� g }g }t|�D ]N}|dkr*|n|}|�t||||| |||| |102d�103� |�t|| | || d�� qt�|�| _t�|�| _ |r�t�t104|d||
dd�g�| _ nd | _ d S �Nr rU r\ T�op��use_convr �padding�name� r` ra rc rd r r r re rf rg r
� downsamplers)rh r r r rQ r r r# r r! rR r& rZ r" r rg rf �irj r9 r: ra � sL 105 ����106 107���zAttnDownBlock2D.__init__Nc C sh d}t | j| j�D ]$\}}|||�}||�}||f7 }q| jd ur`| jD ] }||�}qH||f7 }||fS �Nr9 �rm rg rf r� )rh rn ro �
output_statesrq rp � downsamplerr9 r9 r: rr � s 108 109110111112zAttnDownBlock2D.forward) rL r rM r rN rO Tr rP r T)Nrs r9 r9 rj r: r% � s0 ��=r% c sv e Zd Zdeeeeeeeeeed �113� fd d
�
Zdej e114ej e115ej e116ej e117e ee f e118ej d�dd�Z
� ZS )r' rL r rM r rN rO Tr} rP Fr� c s� t � �� g }g }d| _| | _t|�D ]~}|dkr6|n|}|�t||||| ||||
|119d�120� |s�|�t| || |d| | |||d� � q&|�t| || |d| | d�� q&t �121|�| _ t �122|�| _ |r�t �123t
|d||dd�g�| _nd | _d | _d S )124NTr rU r �r r r( r_ r* r+ r, r r� r� F)r` ra r� r& rc rd r r r r re rf rg r
r� �gradient_checkpointing)rh r r r rQ r r r# r r! rR r& r( rZ r" r r) r* r+ r, rg rf r� rj r9 r: ra � sp 125 ������126 127���z CrossAttnDownBlock2D.__init__N)rn ro r� r� r� r� c C s� d}t | j| j�D ]�\}} | jrp| jrpddd�}128tjj�|129|�||�}tjj�|130| dd�||d d |||�d }n |||�}| |||||d�j}||f7 }q| j d ur�| j D ] } | |�}q�||f7 }||fS ) Nr9 c s � �fdd�}|S )Nc s" �d ur� | d�i�S � | � S d S �N� return_dictr9 ��inputs��moduler� r9 r: �custom_forwardB s zSCrossAttnDownBlock2D.forward.<locals>.create_custom_forward.<locals>.custom_forwardr9 �r� r� r� r9 r� r: �create_custom_forwardA s z;CrossAttnDownBlock2D.forward.<locals>.create_custom_forwardF�r� r r� )N)131rm rg rf �trainingr� r� �utils�132checkpointr� r� ) rh rn ro r� r� r� r� r� rq rp r� r� r9 r9 r: rr 3 s> 133 134� � 135� 136137138139z CrossAttnDownBlock2D.forward)rL r rM r rN rO Tr r} rP r TFFFF)NNNNNr� r9 r9 rj r: r' � sR ��Y ��r' c s@ e Zd Zdeeeeeeeeeed �140� fd141d �
Zdd
d�Z� Z S )r rL r rM r rN rO TrP r� c s� t � �� g }t|�D ]6}|dkr&|n|}|�t||||| |||| |142d�143� qt�|�| _| r|t�t|d||
dd�g�| _ nd | _ d| _144d S )Nr rU Tr� r� F) r` ra rc rd r r re rg r
r� r� )rh r r r rQ r r r# r r! rR rZ r r" rg r� rj r9 r: ra k s8 145 �� 146���zDownBlock2D.__init__Nc C s� d}| j D ]D}| jr:| jr:dd� }tjj�||�||�}n147|||�}||f7 }q148| jd urx| jD ] }||�}q`||f7 }||fS )Nr9 c s � fdd�}|S )Nc s � | � S �Nr9 r� �r� r9 r: r� � s zJDownBlock2D.forward.<locals>.create_custom_forward.<locals>.custom_forwardr9 �r� r� r9 r� r: r� � s z2DownBlock2D.forward.<locals>.create_custom_forward�rg r� r� r� r� r� r� �rh rn ro r� rq r� r� r9 r9 r: rr � s 149 150 151152153154zDownBlock2D.forward)155rL r rM r rN rO TrP Tr )Nrs r9 r9 rj r: r j s. ��3r c s< e Zd Zdeeeeeeeeed � � fd156d �
Zd d
� Z� Z S )r0 rL r rM r rN rO TrP � r r rQ r r r# r r! rR c
s� t � �� g }
t|�D ]6}|dkr&|n|}|
�t||d ||||||157| d�158� qt�|
�| _| r|t�t|d|| dd�g�| _ nd | _ d S )Nr rU Tr� r� )159r` ra rc rd r r re rg r
r� )rh r r rQ r r r# r r! rR rZ r r" rg r� rj r9 r: ra � s6 160 �� 161���zDownEncoderBlock2D.__init__c C s: | j D ]}||d d�}q| jd ur6| jD ] }||�}q(|S �N)ro )rg r� )rh rn rq r� r9 r9 r: rr � s 162163164165zDownEncoderBlock2D.forward)166rL r rM r rN rO TrP Tr rs r9 r9 rj r: r0 � s, ��0r0 c s< e Zd Zdeeeeeeeeed � � fd167d �
Zd d
� Z� Z S )r1 rL r rM r rN rO TrP r� c s� t � �� g }g }t|�D ]N}|dkr*|n|}|�t||d |||||| | d�168� |�t||169| ||d�� qt�|�| _t�|�| _ | r�t�t170|d||
dd�g�| _ nd | _ d S r� r� )rh r r rQ r r r# r r! rR r& rZ r r" rg rf r� rj r9 r: ra � sL 171 ����172 173���z AttnDownEncoderBlock2D.__init__c C sN t | j| j�D ] \}}||d d�}||�}q| jd urJ| jD ] }||�}q<|S r� r� )rh rn rq rp r� r9 r9 r: rr 2 s 174175176177z AttnDownEncoderBlock2D.forward) rL r rM r rN rO Tr rP Tr rs r9 r9 rj r: r1 � s. ��<r1 c sX e Zd Zddddddde�d�ddf178eeeeeeeeed� � fd d179�
Z dd d
�Z180� Z S )r/ rL r rM r rN T� @� r r r rQ r r r# r rR c s t � �� t�g �| _t�g �| _t|�D ]f}|dkr:|n|}| j�t||||t |d d�t |d d�|||| | d� � | j�t181||182| |d�� q*|
r�t||||t |d d�|||| | dddd�
| _ t�t ||d �g�| _
tjd183|d d d �| _nd | _ d | _
d | _d S )
Nr rT rO � r r r rV rW �184groups_outrQ rX rY rZ r[ �r] r^ rV T�fir�
r r r rV rW rQ rX rY rZ r[ �use_in_shortcut�down�kernel�r � �r r �� kernel_size�stride)r` ra r re rf rg rc rd r rb r � resnet_downr r� �Conv2d� skip_conv)rh r r r rQ r r r# r rR r& rZ r" r r� rj r9 r: ra ? sb 185 ���� �z AttnSkipDownBlock2D.__init__Nc C s� d}t | j| j�D ]$\}}|||�}||�}||f7 }q| jd urz| �||�}| jD ] }||�}qT| �|�| }||f7 }|||fS r� )rm rg rf r� r� r� )rh rn ro � skip_sampler� rq rp r� r9 r9 r: rr � s 186 187 188189190zAttnSkipDownBlock2D.forward)NN� rt ru rv �np�sqrtrw rx ry rz ra rr r{ r9 r9 rj r: r/ > s, ��Er/ c sV e Zd Zdddddde�d�ddf eeeeeeeeed� � fd d191�
Z dd d
�Z192� Z S )r. rL r rM r rN Tr� r� c
s� t � �� t�g �| _t|�D ]N}
|
dkr.|n|}| j�t||||t|d d�t|d d�||||193| d� � q | r�t||||t|d d�||||194| dddd�
| _ t�t195||d�g�| _ tj d |d196d197d �| _
nd | _ d | _ d | _
d S ) Nr rT rO r� Tr� r� r� r� r� r� )r` ra r re rg rc rd r rb r� r r� r� r� )rh r r r rQ r r r# r rR rZ r r" r� rj r9 r: ra � sP 198 �� �zSkipDownBlock2D.__init__Nc C sp d}| j D ]}|||�}||f7 }q199| jd urf| �||�}| jD ] }||�}q@| �|�| }||f7 }|||fS r� )rg r� r� r� )rh rn ro r� r� rq r� r9 r9 r: rr � s 200201 202 203204205zSkipDownBlock2D.forward)NNr� r9 r9 rj r: r. � s* ��;r. c s@ e Zd Zdeeeeeeeeeed �206� fd207d �
Zdd
d�Z� Z S )r$ rL r rM r rN rO TrP r� c
s� t � �� g }
t|�D ]6}|dkr&|n|}|
�t||||| |||| |208d�209� qt�|
�| _| r�t�t||||| |||| |210dd� g�| _nd | _d| _ d S )Nr rU T� r r r rV rW rQ rX rY rZ r[ r� F)211r` ra rc rd r r re rg r� r� )rh r r r rQ r r r# r r! rR rZ r rg r� rj r9 r: ra � sL 212 �� ���z ResnetDownsampleBlock2D.__init__Nc C s� d}| j D ]D}| jr:| jr:dd� }tjj�||�||�}n213|||�}||f7 }q214| jd urz| jD ]}|||�}q`||f7 }||fS )Nr9 c s � fdd�}|S )Nc s � | � S r� r9 r� r� r9 r: r� + s zVResnetDownsampleBlock2D.forward.<locals>.create_custom_forward.<locals>.custom_forwardr9 r� r9 r� r: r� * s z>ResnetDownsampleBlock2D.forward.<locals>.create_custom_forwardr� r� r9 r9 r: rr $ s 215 216 217218 219z ResnetDownsampleBlock2D.forward) rL r rM r rN rO TrP T)Nrs r9 r9 rj r: r$ � s, ��<r$ c s@ e Zd Zdeeeeeeeeeed220�221� fd d �
Zddd�Z� Z S )r- rL r rM r rN rO Tr} rP r� c s� t � �� d| _g }g }| | _|| j | _t|�D ]Z}|dkrB|n|}|�t||||| ||||
|222d�223� |�t||| j| | | ddt � d� � q2t224� |�| _ t225� |�| _
|r�t226� t||||| ||||
|227dd� g�| _nd | _d| _d S )NTr rU r� r� F)r` ra r� r& r� rc rd r r228 r r re rf rg r� r� )rh r r r rQ r r r# r r! rR r& r( rZ r rg rf r� rj r9 r: ra @ sp 229 ����
���z#SimpleCrossAttnDownBlock2D.__init__Nc 230 C s� d}|d ur|ni }t | j| j�D ]2\}}|||�}||f||d�|��}||f7 }q"| jd ur�| jD ]} | ||�}qf||f7 }||fS )Nr9 r� r� )231rh rn ro r� r� r� r� rq rp r� r9 r9 r: rr � s$ 232��� 233234 235z"SimpleCrossAttnDownBlock2D.forward) rL r rM r rN rO Tr r} rP T)NNNNrs r9 r9 rj r: r- ? s2 ��S �r- c 236 s< e Zd Zd
eeeeeeeed�� fdd �
Zdd d �Z� ZS )r2 rL rT ��h㈵��>�gelurO F�r r r rQ r r r �resnet_group_sizec237 s� t � �� g }238t|�D ]F} | dkr&|n|}|| } || }
|239�t||||| |
||ddd�240� qt�|241�| _| r�t�t� g�| _ nd | _ d| _242d S )Nr � ada_groupF�243r r rQ r rW r� rV rY rX �conv_shortcut_bias) r` ra rc rd r r re rg r r� r� )rh r r r rQ r r r r� r rg r� rW r� rj r9 r: ra � s2 244 �� zKDownBlock2D.__init__Nc C sv d}| j D ]D}| jr:| jr:dd� }tjj�||�||�}n245|||�}||f7 }q246| jd urn| jD ] }||�}q`||fS )Nr9 c s � fdd�}|S )Nc s � | � S r� r9 r� r� r9 r: r� � s zKKDownBlock2D.forward.<locals>.create_custom_forward.<locals>.custom_forwardr9 r� r9 r� r: r� � s z3KDownBlock2D.forward.<locals>.create_custom_forwardr� r� r9 r9 r: rr � s 247 248 249250251zKDownBlock2D.forward)rL rT r� r� rO F)N) rt ru rv rw rx ry ra rr r{ r9 r9 rj r: r2 � s" ��-r2 c
sB e Zd Zdeeeeeeeeeeed � � fd252d �
Zdd
d�Z� Z S )r3 rL rT rO T�@ Fr� r� ) r r r r( rQ r r� r& r4 r r c
s� t � �� g }
g }d| _t|�D ]j}|dkr0|n|}|| }|| }|
�t||||||| | ddd�253� |�t||| | ||d|254d|d� � q t�|
�| _ t�|�| _255|r�t�t � g�| _ nd | _ d| _
d S )NTr r� Fr� )r( r �attention_biasr4 �cross_attention_norm�256group_size)r` ra r� rc rd r �KAttentionBlockr re rg rf r r� r� )rh r r r r( rQ r r� r r& r4 r r rg rf r� rW r� rj r9 r: ra � sR 257 ���� z KCrossAttnDownBlock2D.__init__Nc C s� d}t | j| j�D ]�\}}| jrf| jrfddd�} tjj�| |�||�}tjj�| |dd�||||�}n |||�}||||||d�}| jd u r�|d7 }q||f7 }q| jd ur�| jD ] }258|259|�}q�||fS ) Nr9 c s � �fdd�}|S )Nc s" �d ur� | d�i�S � | � S d S r� r9 r� r� r9 r: r� @ s zTKCrossAttnDownBlock2D.forward.<locals>.create_custom_forward.<locals>.custom_forwardr9 r� r9 r� r: r� ? s z<KCrossAttnDownBlock2D.forward.<locals>.create_custom_forwardFr� �r� �embr� r� r� )N) rm rg rf r� r� r� r� r� r� ) rh rn ro r� r� r� r� rq rp r� r� r9 r9 r: rr 7 s6 260 261�262�263264 265266267z KCrossAttnDownBlock2D.forward)rL rT rO Tr� Fr� r� )NNNN)268rt ru rv rw rx rz ry ra rr r{ r9 r9 rj r: r3 � s. ��A �r3 c
sB e Zd Zdeeeeeeeeeeed � � fd269d �
Zdd
d�Z� Z S )rB rL r rM r rN rO TrP � r r= r r rQ r r r# r r! rR c s� t � �� g }g }t|�D ]f}||d kr.|n|}|dkr>|n|}|�t|| ||||270||| |
| d�271� |�t|| |
||272d�� qt�|�| _t�|�| _ |r�t�t273|d|d�g�| _ nd | _ d S )Nr r rU r\ T�r� r � r` ra rc rd r r r re rf rg r �274upsamplers)rh r r= r r rQ r r r# r r! rR r& rZ r> rg rf r� �res_skip_channels�resnet_in_channelsrj r9 r: ra g sD 275 ����276 zAttnUpBlock2D.__init__Nc C sr t | j| j�D ]@\}}|d }|d d� }tj||gdd�}|||�}||�}q| jd urn| jD ] }||�}q`|S )N�����r ��dim�rm rg rf r� �catr� )rh rn �res_hidden_states_tuplero rq rp �res_hidden_states� upsamplerr9 r9 r: rr � s 277278279280281zAttnUpBlock2D.forward)282rL r rM r rN rO Tr rP T)Nrs r9 r9 rj r: rB f s0 ��9rB c s� e Zd Zdeeeeeeeeeeed � � fd d
�
Zdej e283ej df e ej e ej e e ee
f e e e ej e ej d�dd�Z� ZS )r@ rL r rM r rN rO Tr} rP F� r r r= r rQ r r r# r r! rR c s t � �� g }g }d| _| | _t|�D ]�}||d kr:|n|}|dkrJ|n|}|�t|| ||||284||| || d�285� |s�|�t| || |d|
|286|||d� � q&|�t| || |d|
|287d�� q&t �288|�| _ t �289|�| _ |r�t �290t
|d|d�g�| _nd | _d| _d S ) NTr r rU r� r r� F)r` ra r� r& rc rd r r r r re rf rg r r� r� )rh r r r= r rQ r r r# r r! rR r& r( rZ r> r) r* r+ r, rg rf r� r� r� rj r9 r: ra � sh 291 ������292 zCrossAttnUpBlock2D.__init__N.)rn r� ro r� r� �
upsample_sizer� r� c C s� t | j| j�D ]�\} }293|d } |d d� }tj|| gdd�}| jr�| jr�d294dd�} tjj�| | �||�}tjj�| |295dd�||d d |||�d }q| ||�}|296|||||d �j }q| j297d ur�| j298D ]}
|
||�}q�|S ) Nr� r r� c s � �fdd�}|S )Nc s" �d ur� | d�i�S � | � S d S r� r9 r� r� r9 r: r� s zQCrossAttnUpBlock2D.forward.<locals>.create_custom_forward.<locals>.custom_forwardr9 r� r9 r� r: r� s z9CrossAttnUpBlock2D.forward.<locals>.create_custom_forwardFr� r r� )N) rm rg rf r� r� r� r� r� r� r� r� )rh rn r� ro r� r� r� r� r� rq rp r� r� r� r9 r9 r: rr s> 299 300� � 301�302303304 zCrossAttnUpBlock2D.forward)rL r rM r rN rO Tr r} rP TFFFF)NNNNNN)rt ru rv rw rx ry rz ra r� r� r r r r rr r{ r9 r9 rj r: r@ � sX ��V � �r@ c
sB e Zd Zdeeeeeeeeeeed � � fd305d �
Zdd
d�Z� Z S )r< rL r rM r rN rO TrP r� c s� t � �� g }t|�D ]N}||d kr*|n|}|dkr:|n|}|�t|| ||||306||| | | d�307� qt�|�| _|
r�t�t|d|d�g�| _ nd | _ d| _308d S )Nr r rU Tr� F) r` ra rc rd r r re rg r r� r� �rh r r= r r rQ r r r# r r! rR rZ r> rg r� r� r� rj r9 r: ra = s0 309 �� zUpBlock2D.__init__Nc C s� | j D ]`}|d }|d d� }tj||gdd�}| jr\| jr\dd� }tjj�||�||�}q|||�}q| jd ur�| jD ]}|||�}qx|S )Nr� r r� c s � fdd�}|S )Nc s � | � S r� r9 r� r� r9 r: r� v s zHUpBlock2D.forward.<locals>.create_custom_forward.<locals>.custom_forwardr9 r� r9 r� r: r� u s z0UpBlock2D.forward.<locals>.create_custom_forward�rg r� r� r� r� r� r� r� � rh rn r� ro r� rq r� r� r� r9 r9 r: rr l s 310 311312 zUpBlock2D.forward) rL r rM r rN rO TrP T)NNrs r9 r9 rj r: r< < s. ��/r< c s< e Zd Zdeeeeeeeeed � � fd313d �
Zd d
� Z� Z S )rE rL r rM r rN rO TrP r� c s� t � �� g } t|�D ]6}
|
dkr&|n|}| �t||d ||||||314| d�315� qt�| �| _| rxt�t|d|d�g�| _ nd | _ d S )Nr rU Tr� )316r` ra rc rd r r re rg r r� )rh r r rQ r r r# r r! rR rZ r> rg r� �input_channelsrj r9 r: ra � s, 317 �� zUpDecoderBlock2D.__init__c C s: | j D ]}||d d�}q| jd ur6| jD ] }||�}q(|S r� )rg r� )rh rn rq r� r9 r9 r: rr � s 318319320321zUpDecoderBlock2D.forward) rL r rM r rN rO TrP Trs r9 r9 rj r: rE � s* ��*rE c s< e Zd Zdeeeeeeeeed � � fd322d �
Zd d
� Z� Z S )rF rL r rM r rN rO TrP r� c
s� t � �� g }
g }t|�D ]N}|dkr*|n|}|
�t||d |||||| | d�323� |�t||324| ||d�� qt�|�| _t�|
�| _ | r�t�t325|d|d�g�| _ nd | _ d S )Nr rU r\ Tr� r� )rh r r rQ r r r# r r! rR r& rZ r> rg rf r� r� rj r9 r: ra � sB 326 ����327 z AttnUpDecoderBlock2D.__init__c C sN t | j| j�D ] \}}||d d�}||�}q| jd urJ| jD ] }||�}q<|S r� )rm rg rf r� )rh rn rq rp r� r9 r9 r: rr � s 328329330331z AttnUpDecoderBlock2D.forward)332rL r rM r rN rO Tr rP Trs r9 r9 rj r: rF � s, ��6rF c sZ e Zd Zddddddde�d�ddf333eeeeeeeeeed�334� fd d335�
Z dd d
�Z336� Z S )rD rL r rM r rN Tr� �337r r= r r rQ r r r# r rR c sZ t � �� t�g �| _t�g �| _t|�D ]j}||d kr>|n|}|dkrN|n|}| j�t|| |||t ||d d�t |d d�||| | |338d� � q*| j�t339|| | |d�� t ||d�| _ |�r>t||||t |d d�t |d d�||| | |340ddd d341�| _
tj|d d d
d
d�| _tjjt |d d�||dd�| _t�� | _nd | _
d | _d | _d | _d S )Nr r rT rO r� r� r� Tr� �r r r rV rW r� rQ rX rY rZ r[ r� �upr� r� �r� r� r� �r� r� r� ��342num_groups� num_channelsrV �affine)r` ra r re rf rg rc rd r rb r r r� � resnet_upr� r� r� � GroupNorm� skip_norm�SiLU�act)rh r r= r r rQ r r r# r rR r& rZ �upsample_paddingr> r� r� r� rj r9 r: ra sp 343 ���� �� zAttnSkipUpBlock2D.__init__Nc C s� | j D ]4}|d }|d d� }tj||gdd�}|||�}q| jd |�}|d ur^| �|�}nd}| jd ur�| �|�}| �|�}| �|�}|| }| �||�}||fS �Nr� r r� r ) rg r� r� rf r� r� r� r r� �rh rn r� ro r� rq r� Zskip_sample_statesr9 r9 r: rr O s 344 345346347348 zAttnSkipUpBlock2D.forward)NNr� r9 r9 rj r: rD � s. ��OrD c sX e Zd Zdddddde�d�ddf eeeeeeeeeed�349� fd d350�
Z dd d
�Z351� Z S )rC rL r rM r rN Tr� r� c s6 t � �� t�g �| _t|�D ]j}||d kr2|n|}|dkrB|n|}| j�t|| |||t|| d d�t|d d�||| | |352d� � q t ||d�| _353| �rt||||t|d d�t|d d�||| | |354dddd �| _ tj |d355d d d d
�| _
tjjt|d d�||dd�| _t�� | _nd | _ d | _
d | _d | _d S )Nr r rT rO r� r� Tr� r� r� r� r� r� r� )r` ra r re rg rc rd r rb r r� r� r� r� r� r� r� r� r )rh r r= r r rQ r r r# r rR rZ r> r r� r� r� rj r9 r: ra l s^ 356 �� �� zSkipUpBlock2D.__init__Nc C s� | j D ]4}|d }|d d� }tj||gdd�}|||�}q|d urP| �|�}nd}| jd ur�| �|�}| �|�}| �|�}|| }| �||�}||fS r )rg r� r� r� r� r� r r� r r9 r9 r: rr � s 357 358359360361 zSkipUpBlock2D.forward)NNr� r9 r9 rj r: rC k s, ��DrC c
sB e Zd Zdeeeeeeeeeeed � � fd362d �
Zdd
d�Z� Z S )r? rL r rM r rN rO TrP r� c s� t � �� g }t|�D ]N}||d kr*|n|}|dkr:|n|}|�t|| ||||363||| | | d�364� qt�|�| _|
r�t�t|||||365||| | | dd� g�| _nd | _d| _ d S )Nr r rU T� r r r rV rW rQ rX rY rZ r[ r� F)366r` ra rc rd r r re rg r� r� r� rj r9 r: ra � sN 367 �� ���z ResnetUpsampleBlock2D.__init__Nc C s� | j D ]`}|d }|d d� }tj||gdd�}| jr\| jr\dd� }tjj�||�||�}q|||�}q| jd ur�| jD ]}|||�}qx|S )Nr� r r� c s � fdd�}|S )Nc s � | � S r� r9 r� r� r9 r: r� s zTResnetUpsampleBlock2D.forward.<locals>.create_custom_forward.<locals>.custom_forwardr9 r� r9 r� r: r� s z<ResnetUpsampleBlock2D.forward.<locals>.create_custom_forwardr� r� r9 r9 r: rr 368 s 369 370371 z ResnetUpsampleBlock2D.forward) rL r rM r rN rO TrP T)NNrs r9 r9 rj r: r? � s. ��?r? c
sB e Zd Zdeeeeeeeeeeed372� � fd d �
Zddd�Z� Z S )rA rL r rM r rN rO Tr} rP r� c s� t � �� g }g }d| _| | _|| j | _t|�D ]r}||d krF|n|}|dkrV|n|}|�t|| ||||373||| || d�374� |�t||| j| |
|375ddt � d� � q2t376� |�| _ t377� |�| _
|r�t378� t|||||379||| || dd� g�| _nd | _d| _d S )NTr r rU r� r F)r` ra r� r& r� rc rd r r380 r r re rf rg r� r� )rh r r r= r rQ r r r# r r! rR r& r( rZ r> rg rf r� r� r� rj r9 r: ra % sr 381 ����
���z!SimpleCrossAttnUpBlock2D.__init__Nc C s� |d ur |ni }t | j| j�D ]N\}} |d }382|d d� }tj||383gdd�}|||�}| |f||d�|��}q | jd ur�| jD ]} | ||�}q~|S )Nr� r r� r� r� ) rh rn r� ro r� r� r� r� rq rp r� r� r9 r9 r: rr y s$ 384 385���386387 z SimpleCrossAttnUpBlock2D.forward) rL r rM r rN rO Tr r} rP T)NNNNNrs r9 r9 rj r: rA $ s< ��X �rA c 388 s@ e Zd Zd
eeeeeeeee d�� fdd �
Zdd d �Z� Z S )rG rL � r� r� rO Tr� c389 s� t � �� g }390d| } |} |d }t|�D ]V}
|
dkr:| n|}|| }|| }|391�t||
|d krf| n|||||||ddd�392� q*t�|393�| _| r�t�t� g�| _ nd | _ d| _394d S )N� r r r� F)395r r r rV rW r� rQ rY rX r� ) r` ra rc rd r r re rg r r� r� )rh r r r rQ r r r r� r> rg �
k_in_channels�k_out_channelsr� rW r� rj r9 r: ra � s8 396 �� zKUpBlock2D.__init__Nc C s� |d }|d ur"t j||gdd�}| jD ]:}| jrX| jrXdd� }t jj�||�||�}q(|||�}q(| jd ur�| jD ] }||�}qt|S )Nr� r r� c s � fdd�}|S )Nc s � | � S r� r9 r� r� r9 r: r� � s zIKUpBlock2D.forward.<locals>.create_custom_forward.<locals>.custom_forwardr9 r� r9 r� r: r� � s z1KUpBlock2D.forward.<locals>.create_custom_forward)r� r� rg r� r� r� r� r� )rh rn r� ro r� rq r� r� r9 r9 r: rr � s 397 398399400zKUpBlock2D.forward)rL r r� r� rO T)NN)401rt ru rv rw rx ry r ra rr r{ r9 r9 rj r: rG � s" ��/rG c
sB e Zd Zdeeeeeeeeeeed402� � fd d �
Zddd�Z� Z S )rH rL rT r� r� rO r � TF) r r r rQ r r r r� r( r> r, c
sZ t � �� g }
g }|| ko$|kn }||k}|r:dnd}d| _| | _|rR|nd| }|}|d }t|�D ]�}|dkr~|n|}|| }|| }|r�||d kr�|}nd }|
�t|||||||||ddd� � |�t||d kr�|n|||d kr�|| n|| | |403|d|d| d� � qnt� |
�| _404t� |�| _ | �rJt� t � g�| _
nd | _
d| _d S ) NTFr r r r� ) r r �conv_2d_out_channelsr rV rW r� rQ rY rX r� )r( r r� r4 r� r, )r` ra r� r& rc rd r r� r re rg rf r r� r� )rh r r r rQ r r r r� r& r( r> r, rg rf Zis_first_blockZis_middle_blockr4 r r r� rW r� r405 rj r9 r: ra � sl 406 ��407�408�� z KCrossAttnUpBlock2D.__init__Nc C s� |d }|d ur"t j||gdd�}t| j| j�D ]t\}} | jr�| jr�d409dd�}410t jj�|411|�||�}t jj�|412| dd�||||�d }q0|||�}| |||||d �}q0| j d ur�| j D ] } | |�}q�|S ) Nr� r r� c s � �fdd�}|S )Nc s" �d ur� | d�i�S � | � S d S r� r9 r� r� r9 r: r� K413 s zRKCrossAttnUpBlock2D.forward.<locals>.create_custom_forward.<locals>.custom_forwardr9 r� r9 r� r: r� J414 s z:KCrossAttnUpBlock2D.forward.<locals>.create_custom_forwardFr� r r� )N)415r� r� rm rg rf r� r� r� r� r� ) rh rn r� ro r� r� r� r� rq rp r� r� r9 r9 r: rr 9416 s8 417 418 419��420�421422423zKCrossAttnUpBlock2D.forward) rL rT r� r� rO r r TF)NNNNNrs r9 r9 rj r: rH � s8 ��W �rH c
sZ e Zd ZdZdeeeeee eeeeeed� � fdd �
Zd424d � Z d d
� Z425ddd�Z � Z S )r� a~ 426 A basic Transformer block.427 428 Parameters:429 dim (`int`): The number of channels in the input and output.430 num_attention_heads (`int`): The number of heads to use for multi-head attention.431 attention_head_dim (`int`): The number of channels in each head.432 dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use.433 cross_attention_dim (`int`, *optional*): The size of the encoder_hidden_states vector for cross attention.434 activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward.435 num_embeds_ada_norm (:436 obj: `int`, *optional*): The number of diffusion steps used during training. See `Transformer2DModel`.437 attention_bias (:438 obj: `bool`, *optional*, defaults to `False`): Configure if the attentions should contain a bias parameter.439 rL NFr rO ) r� �num_attention_heads�attention_head_dimrQ r( r� r, r r4 r� r� c 440 sz t � �� | | _| rDt||td|| ��| _t|||||d dd�| _t||td|| ��| _t||||||||441d�| _ d S )Nr F)r� r� r� rQ r� r( r� )r� r( r� r� rQ r� r, r� )442r` ra r4 r �max�norm1r443 �attn1�norm2�attn2) rh r� r r rQ r( r� r, r r4 r� r� rj r9 r: ra ~444 s0 445� �zKAttentionBlock.__init__c C s$ |� dddd��|jd || d�S )Nr r r� r r� ��permute�reshape�shape�rh rn �height�weightr9 r9 r: �_to_3d�446 s zKAttentionBlock._to_3dc C s |� ddd��|jd d||�S )Nr r r r� r r r9 r9 r: �_to_4d�447 s zKAttentionBlock._to_4dc 448 C s� |d ur |ni }| j rn| �||�}|jdd � \}}| �|||�}| j|fdd i|��} | �| ||�} | | }| �||�}|jdd � \}}| �|||�}| j|fd|i|��} | �| ||�} | | }|S )Nr r� )r4 r r r r r r r )449rh rn r� r� r� r� Znorm_hidden_statesr r � attn_outputr9 r9 r: rr �450 s6 ��� ���zKAttentionBlock.forward)rL NFFr FFrO )NNNN)
rt ru rv �__doc__rw rx r rz ra r r rr r{ r9 r9 rj r: r� m451 s: ��+ �r� )NNNFFFFr )NNFFFFr )9�typingr r r r �numpyr� r� r Z attentionr r Zattention_processorr452 r �dual_transformer_2dr rq r
r r r r r r �transformer_2dr r r; rJ �ModulerK r| r� r% r' r r0 r1 r/ r. r$ r- r2 r3 rB r@ r< rE rF rD rC r? rA rG rH r� r9 r9 r9 r: �<module> sp $
�453 8 �454 -Om^O O<IZOXpGpK J6Cl_ZxI