CoolFace
Apppublic

declare-lab/tango2

sourceHugging Faceupdated 2y agoView on Hugging Face
92likes
rootdiffuserssrcdiffusersmodels__pycache__unet_2d_condition.cpython-39.pyc
unet_2d_condition.cpython-39.pyc233 linesDownload Raw Back to __pycache__
1a

2�0dn��@sddlmZddlmZmZmZmZmZmZddl	Z	ddl3mZddlZ	ddl
mZmZddlmZddlmZmZdd	lmZmZdd4lmZmZmZddlmZddlmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&e�'e(�Z)eGd
d�de��Z*Gdd�deee�Z+dS)�)�	dataclass)�Any�Dict�List�Optional�Tuple�UnionN�)�ConfigMixin�register_to_config)�UNet2DConditionLoadersMixin)�5BaseOutput�logging�)�AttentionProcessor�
AttnProcessor)�GaussianFourierProjection�TimestepEmbedding�	Timesteps)�6ModelMixin)�CrossAttnDownBlock2D�CrossAttnUpBlock2D�DownBlock2D�UNetMidBlock2DCrossAttn�UNetMidBlock2DSimpleCrossAttn�	UpBlock2D�get_down_block�get_up_blockc@seZdZUdZejed<dS)�UNet2DConditionOutputz�7    Args:8        sample (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`):9            Hidden states conditioned on `encoder_hidden_states` input. Output of last layer of model.10    �sampleN)�__name__�11__module__�__qualname__�__doc__�torch�FloatTensor�__annotations__�r'r'�W/home/deep/Projects/audio_diffusion/diffusers/src/diffusers/models/unet_2d_condition.pyr*s12rc#s�eZdZdZdZed'eeeeeeee	e13ee14e	e15eee	efe	eeeee16eeeeee	efeee	efeeee17eeee18e19ee20eeeeeeed� �fdd�
�Z
eee21efd�dd��Zeeee22effd�dd�Zdd�Zd d!�Zd(d"d#�Zd)ejeejeefejeejeejeejeee23efee	ejeejeejeeee	fd$�d%d&�Z�ZS)*�UNet2DConditionModela�24    UNet2DConditionModel is a conditional 2D UNet model that takes in a noisy sample, conditional state, and a timestep25    and returns sample shaped output.26 27    This model inherits from [`ModelMixin`]. Check the superclass documentation for the generic methods the library28    implements for all the models (such as downloading or saving, etc.)29 30    Parameters:31        sample_size (`int` or `Tuple[int, int]`, *optional*, defaults to `None`):32            Height and width of input/output sample.33        in_channels (`int`, *optional*, defaults to 4): The number of channels in the input sample.34        out_channels (`int`, *optional*, defaults to 4): The number of channels in the output.35        center_input_sample (`bool`, *optional*, defaults to `False`): Whether to center the input sample.36        flip_sin_to_cos (`bool`, *optional*, defaults to `False`):37            Whether to flip the sin to cos in the time embedding.38        freq_shift (`int`, *optional*, defaults to 0): The frequency shift to apply to the time embedding.39        down_block_types (`Tuple[str]`, *optional*, defaults to `("CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "DownBlock2D")`):40            The tuple of downsample blocks to use.41        mid_block_type (`str`, *optional*, defaults to `"UNetMidBlock2DCrossAttn"`):42            The mid block type. Choose from `UNetMidBlock2DCrossAttn` or `UNetMidBlock2DSimpleCrossAttn`, will skip the43            mid block layer if `None`.44        up_block_types (`Tuple[str]`, *optional*, defaults to `("UpBlock2D", "CrossAttnUpBlock2D", "CrossAttnUpBlock2D", "CrossAttnUpBlock2D",)`):45            The tuple of upsample blocks to use.46        only_cross_attention(`bool` or `Tuple[bool]`, *optional*, default to `False`):47            Whether to include self-attention in the basic transformer blocks, see48            [`~models.attention.BasicTransformerBlock`].49        block_out_channels (`Tuple[int]`, *optional*, defaults to `(320, 640, 1280, 1280)`):50            The tuple of output channels for each block.51        layers_per_block (`int`, *optional*, defaults to 2): The number of layers per block.52        downsample_padding (`int`, *optional*, defaults to 1): The padding to use for the downsampling convolution.53        mid_block_scale_factor (`float`, *optional*, defaults to 1.0): The scale factor to use for the mid block.54        act_fn (`str`, *optional*, defaults to `"silu"`): The activation function to use.55        norm_num_groups (`int`, *optional*, defaults to 32): The number of groups to use for the normalization.56            If `None`, it will skip the normalization and activation layers in post-processing57        norm_eps (`float`, *optional*, defaults to 1e-5): The epsilon to use for the normalization.58        cross_attention_dim (`int` or `Tuple[int]`, *optional*, defaults to 1280):59            The dimension of the cross attention features.60        attention_head_dim (`int`, *optional*, defaults to 8): The dimension of the attention heads.61        resnet_time_scale_shift (`str`, *optional*, defaults to `"default"`): Time scale shift config62            for resnet blocks, see [`~models.resnet.ResnetBlock2D`]. Choose from `default` or `scale_shift`.63        class_embed_type (`str`, *optional*, defaults to None):64            The type of class embedding to use which is ultimately summed with the time embeddings. Choose from `None`,65            `"timestep"`, `"identity"`, `"projection"`, or `"simple_projection"`.66        num_class_embeds (`int`, *optional*, defaults to None):67            Input dimension of the learnable embedding matrix to be projected to `time_embed_dim`, when performing68            class conditioning with `class_embed_type` equal to `None`.69        time_embedding_type (`str`, *optional*, default to `positional`):70            The type of position embedding to use for timesteps. Choose from `positional` or `fourier`.71        timestep_post_act (`str, *optional*, default to `None`):72            The second activation function to use in timestep embedding. Choose from `silu`, `mish` and `gelu`.73        time_cond_proj_dim (`int`, *optional*, default to `None`):74            The dimension of `cond_proj` layer in timestep embedding.75        conv_in_kernel (`int`, *optional*, default to `3`): The kernel size of `conv_in` layer.76        conv_out_kernel (`int`, *optional*, default to `3`): The kernel size of `conv_out` layer.77        projection_class_embeddings_input_dim (`int`, *optional*): The dimension of the `class_labels` input when78            using the "projection" `class_embed_type`. Required when using the "projection" `class_embed_type`.79        class_embeddings_concat (`bool`, *optional*, defaults to `False`): Whether to concatenate the time80        embeddings with the class embeddings.81    TN�Fr�rrrrr�rrrr�i@i��r.r	r�silu� ��h㈵��>r.��default�82positional�) �sample_size�in_channels�out_channels�center_input_sample�flip_sin_to_cos�83freq_shift�down_block_types�mid_block_type�up_block_types�only_cross_attention�block_out_channels�layers_per_block�downsample_padding�mid_block_scale_factor�act_fn�norm_num_groups�norm_eps�cross_attention_dim�attention_head_dim�dual_cross_attention�use_linear_projection�class_embed_type�num_class_embeds�upcast_attention�resnet_time_scale_shift�time_embedding_type�timestep_post_act�time_cond_proj_dim�conv_in_kernel�conv_out_kernel�%projection_class_embeddings_input_dim�class_embeddings_concatc!3s:t���||_t|�t|	�kr6td|�d|	�d���t|�t|�kr\td|�d|�d���t|84t�s�t|85�t|�kr�td|86�d|�d���t|t�s�t|�t|�kr�td|�d|�d���t|t�r�t|�t|�kr�td|�d|�d���|d	d87}!t	j88||d||!d�|_|d
k�r`|dd89}"|"d90dk�rDtd|"�d���t|"d91dd|d�|_
|"}#n@|dk�r�|dd}"t|d||�|_
|d}#nt|�d���t|#|"|||d�|_|du�r�|du�r�t	�||"�|_n�|dk�r�t|#|"�|_nv|dk�r92t	�|"|"�|_n\|dk�r4|du�r&td��t||"�|_n2|dk�r`|du�rPtd��t	�||"�|_nd|_t	�g�|_t	�g�|_t|93t��r�|94gt|�}95t|t��r�|ft|�}t|t��r�|ft|�}| �r�|"d96}$n|"}$|d}%t|�D]j\}&}'|%}(||&}%|&t|�d	k})t|'||(|%|$|)|||||&||&|
|||97|&||d�}*|j�|*��q�|dk�r�t|d|$|||||d|d||||d�|_nT|dk�r�t|d|$||||d|d||d �	|_n |du�r�d|_ntd!|����d|_tt|��}+tt|��},tt|��}-tt|98��}99|+d}%t|	�D]�\}&}.|&t|�d	k})|%}/|+|&}%|+t |&d	t|�d	�}(|)�s�d"}0|jd	7_nd}0t!|.|d	|(|%|/|$|0||||-|&|,|&|||100|&||d#�}1|j�|1�|%}/�q0|du�rt	j"|d||d$�|_#t	�$�|_%nd|_#d|_%|d	d101}2t	j102|d|||2d�|_&dS)%Nz\Must provide the same number of `down_block_types` as `up_block_types`. `down_block_types`: z. `up_block_types`: �.zbMust provide the same number of `block_out_channels` as `down_block_types`. `block_out_channels`: z. `down_block_types`: zfMust provide the same number of `only_cross_attention` as `down_block_types`. `only_cross_attention`: zbMust provide the same number of `attention_head_dim` as `down_block_types`. `attention_head_dim`: zdMust provide the same number of `cross_attention_dim` as `down_block_types`. `cross_attention_dim`: rr	r)�kernel_size�paddingZfourierz2`time_embed_dim` should be divisible by 2, but is F)�set_W_to_weight�logr:r4r*zJ does not exist. Please make sure to use one of `fourier` or `positional`.)rD�post_act_fn�
cond_proj_dim�timestep�identity�103projectionzX`class_embed_type`: 'projection' requires `projection_class_embeddings_input_dim` be setZsimple_projectionz_`class_embed_type`: 'simple_projection' requires `projection_class_embeddings_input_dim` be set)�104num_layersr7r8�
temb_channels�add_downsample�105resnet_eps�
resnet_act_fn�
resnet_groupsrG�attn_num_head_channelsrBrIrJr?rMrNr�����)r7rarcrd�output_scale_factorrNrGrfrerIrJrMr)	r7rarcrdrhrGrfrerNzunknown mid_block_type : T)r`r7r8�prev_output_channelra�add_upsamplercrdrerGrfrIrJr?rMrN)�num_channels�106num_groups�eps)'�super�__init__r6�len�107ValueError�108isinstance�bool�int�list�nn�Conv2d�conv_inr�	time_projrr�time_embedding�	Embedding�class_embedding�Identity�Linear�109ModuleList�down_blocks�	up_blocks�	enumerater�appendr�	mid_blockr�num_upsamplers�reversed�minr�	GroupNorm�
conv_norm_out�SiLU�conv_act�conv_out)3�selfr6r7r8r9r:r;r<r=r>r?r@rArBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQrRrSrTrU�conv_in_padding�time_embed_dim�timestep_input_dimZblocks_time_embed_dim�output_channel�i�down_block_type�
input_channel�is_final_block�110down_block�reversed_block_out_channelsZreversed_attention_head_dimZreversed_cross_attention_dim�
up_block_typerirj�up_blockZconv_out_padding��	__class__r'r(rotsh)111������112�113114��	115116117118�119120121�122�123�124125�126127�128129��zUNet2DConditionModel.__init__)�returncsHi}ttjjtttfd��fdd��|��D]\}}�|||�q.|S)z�130        Returns:131            `dict` of attention processors: A dictionary containing all attention processors used in the model with132            indexed by its weight name.133        )�name�module�134processorscsFt|d�r|j||�d�<|��D]\}}�|�d|��||�q"|S�N�
set_processorz135.processorrV)�hasattr�	processor�named_children)r�r�r��sub_name�child��fn_recursive_add_processorsr'r(r��s136137zIUNet2DConditionModel.attn_processors.<locals>.fn_recursive_add_processors)�strr$rv�Modulerrr�)r�r�r�r�r'r�r(�attn_processors�s138"	z$UNet2DConditionModel.attn_processors)r�cs~t|j���}t|t�rDt|�|krDtdt|��d|�d|�d���ttjj	d��fdd��|�139�D]\}}�|||�qddS)	a�140        Parameters:141            `processor (`dict` of `AttentionProcessor` or `AttentionProcessor`):142                The instantiated processor class or a dictionary of processor classes that will be set as the processor143                of **all** `Attention` layers.144            In case `processor` is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors.:145 146        z>A dict of processors was passed, but the number of processors z0 does not match the number of attention layers: z. Please make sure to pass z processor classes.)r�r�csbt|d�r6t|t�s |�|�n|�|�|�d���|��D]\}}�|�d|��||�q>dSr�)r�rr�dictr��popr�)r�r�r�r�r���fn_recursive_attn_processorr'r(r��s147148zLUNet2DConditionModel.set_attn_processor.<locals>.fn_recursive_attn_processorN)rpr��keysrrr�rqr�r$rvr�r�)r�r��countr�r�r'r�r(�set_attn_processor�s	���149z'UNet2DConditionModel.set_attn_processorcCs|�t��dS)ze150        Disables custom attention processors and sets the default attention implementation.151        N)r�r)r�r'r'r(�set_default_attn_processor�sz/UNet2DConditionModel.set_default_attn_processorc152sJg�tjjd���fdd��|��D]}�|�q$t��}|dkrRdd��D�}n|dkrd|dg}t|t�sx||gn|}t|�t��kr�td	t|��d153|j�dt���dt���d
�	��t	t|��D]:}||}�|}|dur�||kr�td|�d|�d
���q�tjjt154td��fdd��tt|��}|��D]}�||��q4dS)a155        Enable sliced attention computation.156 157        When this option is enabled, the attention module will split the input tensor in slices, to compute attention158        in several steps. This is useful to save some memory in exchange for a small speed decrease.159 160        Args:161            slice_size (`str` or `int` or `list(int)`, *optional*, defaults to `"auto"`):162                When `"auto"`, halves the input to the attention heads, so attention will be computed in two steps. If163                `"max"`, maximum amount of memory will be saved by running only one slice at a time. If a number is164                provided, uses as many slices as `attention_head_dim // slice_size`. In this case, `attention_head_dim`165                must be a multiple of `slice_size`.166        )r�cs0t|d�r��|j�|��D]}�|�qdS�N�set_attention_slice)r�r��sliceable_head_dim�children)r�r�)�$fn_recursive_retrieve_sliceable_dims�sliceable_head_dimsr'r(r��s167zVUNet2DConditionModel.set_attention_slice.<locals>.fn_recursive_retrieve_sliceable_dims�autocSsg|]}|d�qS)r	r')�.0�dimr'r'r(�168<listcomp>��z<UNet2DConditionModel.set_attention_slice.<locals>.<listcomp>�maxrzYou have provided z, but z has zH different attention layers. Make sure to match `len(slice_size)` to be rVNzsize z has to be smaller or equal to )r��169slice_sizecs4t|d�r|�|���|��D]}�||�q dSr�)r�r�r�r�)r�r�r�)� fn_recursive_set_attention_slicer'r(r��s170zRUNet2DConditionModel.set_attention_slice.<locals>.fn_recursive_set_attention_slice)
r$rvr�r�rprrrurq�config�rangerrtr�)r�r�r��num_sliceable_layersr��sizer��reversed_slice_sizer')r�r�r�r(r��s2171172��z(UNet2DConditionModel.set_attention_slicecCst|ttttf�r||_dS)N)rrrrrr�gradient_checkpointing)r�r��valuer'r'r(�_set_gradient_checkpointingsz0UNet2DConditionModel._set_gradient_checkpointing)rr]�encoder_hidden_states�class_labels�
timestep_cond�attention_mask�cross_attention_kwargs�down_block_additional_residuals�mid_block_additional_residual�encoder_attention_mask�return_dictr�cs�d|j�d}d}
t�fdd�|jdd�D��r@t�d�d}|durfd	|�|j�d173}|�d	�}|174dur�|175jtj	ur�d	|176�|j�d177}178|179�d	�}180|j181jr�d|d}|}t�|��s182|j
jdk}t|t�r�|r�tjntj}n|r�tjntj}tj|g||j
d
�}n t|j�dk�r*|d�|j
�}|�|jd�}|�|�}|j|jd�}|�||�}|jdu�r�|du�r|td��|j183jdk�r�|�|�}|�|�j|jd�}|j184j�r�tj||gdd�}n||}|�|�}|f}|j D]L}t!|d��r|j"�r|||||||185d�\}}n|||d�\}}||7}�q�|du�rld}t#||�D]\}}||}||f7}�qJ|}|j$du�r�|j$||||||186d�}|	du�r�||	}t%|j&�D]�\}}|t|j&�d	k}|t|j'�d�}|dt|j'��}|�s187|�r188|djdd�}
t!|d��r8|j"�r8|||||||
||189d�}n|||||
d�}�q�|j(�rh|�(|�}|�)|�}|�*|�}|�s~|fSt+|d�S)a�190        Args:191            sample (`torch.FloatTensor`): (batch, channel, height, width) noisy inputs tensor192            timestep (`torch.FloatTensor` or `float` or `int`): (batch) timesteps193            encoder_hidden_states (`torch.FloatTensor`): (batch, sequence_length, feature_dim) encoder hidden states194            encoder_attention_mask (`torch.Tensor`):195                (batch, sequence_length) cross-attention mask (or bias), applied to encoder_hidden_states. If a196                BoolTensor is provided, it will be turned into a bias, by adding a large negative value. False = hide197                token. Other tensor types will be used as-is as bias values.198            return_dict (`bool`, *optional*, defaults to `True`):199                Whether or not to return a [`models.unet_2d_condition.UNet2DConditionOutput`] instead of a plain tuple.200            cross_attention_kwargs (`dict`, *optional*):201                A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under202                `self.processor` in203                [diffusers.cross_attention](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/cross_attention.py).204 205        Returns:206            [`~models.unet_2d_condition.UNet2DConditionOutput`] or `tuple`:207            [`~models.unet_2d_condition.UNet2DConditionOutput`] if `return_dict` is True, otherwise a `tuple`. When208            returning a tuple, the first element is the sample tensor.209        r	FNc3s|]}|�dkVqdS)rNr')r��s�Zdefault_overall_up_factorr'r(�	<genexpr>5r�z/UNet2DConditionModel.forward.<locals>.<genexpr>�����z9Forward upsample size to force interpolation output size.Trg���g�?�mps)�dtype�devicer)r�z9class_labels should be provided when num_class_embeds > 0r]rg)r��has_cross_attention)�
hidden_states�tembr�r�r�r�)r�r�r')r�r�r�r�)r�r��res_hidden_states_tupler�r��
upsample_sizer�r�)r�r�r�r�)r),r��any�shape�logger�info�tor��	unsqueezer$rsr�r9�	is_tensorr��typerr�float�float32�float64�int32�int64�tensorrp�expandryrzr|rqrKrU�catrxr�r�r��zipr�r�r��resnetsr�r�r�r)r�rr]r�r�r�r�r�r�r�r�r�Zforward_upsample_sizer��	timesteps�is_mpsr��t_emb�emb�	class_emb�down_block_res_samples�downsample_block�res_samplesZnew_down_block_res_samples�down_block_res_sampleZdown_block_additional_residualr�Zupsample_blockr�r'r�r(�forwards�'210 211212213214215216217218219220�	221��	222��223224225226zUNet2DConditionModel.forward) Nr*r*FTrr+rr,Fr-r	rrr/r0r1r.r2FFNNFr3r4NNr5r5NF)F)NNNNNNNT)r r!r"r#� _supports_gradient_checkpointingrrrtrsrr�rr�ro�propertyrrr�r�r�r�r�r$r%�Tensorrrr��
__classcell__r'r'r�r(r)5s�<��A227	�228�r)),�dataclassesr�typingrrrrrrr$�torch.nnrvZtorch.utils.checkpoint�configuration_utilsr229r�loadersr�utilsr
r�attention_processorrr�230embeddingsrrr�modeling_utilsr�unet_2d_blocksrrrrrrrr�231get_loggerr r�rr)r'r'r'r(�<module>s (232233