RuiTerrty/RemoteSensingChangeDetection-RSCD.HA2F
0
1a
2 8��g�% � @ s� U d dl Z d dlZd dlmZmZmZmZmZ d dlZd dl Z d dl m3Z4m Z ddl m
Z
mZ ddlmZ ddlmZ ddlmZ e �d �Zej�d5�du Zz:er�d d lmZm Z m Z d Z e� d
� ne� d� e �W n e y� dZ e� d� Y n0 G dd� de6j!�Z"d#e ee ge f e#e d�dd�Z$d$dd�Z%d%dd�Z&i Z'eeef e(d< d&d d �Z)d'ee ee ege f e#e d �d d �Z*G d!d"� d"e"�Z+dS )(� N)�Callable�List�Any�Tuple�Dict)�nn�Tensor� )� Attention�MemEffAttention)�DropPath)�7LayerScale)�MlpZdinov2�XFORMERS_DISABLED)�fmha�scaled_index_add�index_select_catTz xFormers is available (Block)z xFormers is disabled (Block)Fz!xFormers is not available (Block)c s� e Zd Zddddddddejejeef eee e8e9e10e e e e dej f e dej f e dej f e dej f dd�� fdd �
Z
eed11�d d �Z� ZS )
�Blockg @FT� N.)�dim� num_heads� mlp_ratio�qkv_bias� proj_bias�ffn_bias�drop� attn_drop� drop_path� act_layer�12norm_layer�13attn_class� ffn_layer�returnc s� t � �� | |�| _|
||||||d�| _| r:t|| d�nt�� | _|14dkrTt|15�nt�� | _ | |�| _16t || �}|||| ||d�| _ | r�t|| d�nt�� | _
|17dkr�t|18�nt�� | _|19| _d S )N)r r r r Z proj_drop)� init_valuesr )� in_features�hidden_featuresr r �bias)�super�__init__�norm1�attnr
r �Identity�ls1r �20drop_path1�norm2�int�mlp�ls2Z21drop_path2�sample_drop_ratio)�selfr r r r r r r r r# r r r r r! Zmlp_hidden_dim�� __class__� �BC:\Users\82607\Desktop\code\ChangeViT-master\model\layers\block.pyr( , s0 2223�24 �zBlock.__init__��xr" c s� t t d�� fdd� }t t d�� fdd� }� jrZ� jdkrZt||� jd�}t||� jd�}nN� jr�� jdkr�|� �||�� }|� �||�� }n|||� }|||� }|S ) Nr8 c s � � � �� �| ���S �N�r, r* r) �r9 �r3 r6 r7 �attn_residual_funcZ s z)Block.forward.<locals>.attn_residual_funcc s � � � �� �| ���S r: �r1 r0 r. r< r= r6 r7 �ffn_residual_func] s z(Block.forward.<locals>.ffn_residual_funcg�������?)�
residual_funcr2 r )r �trainingr2 �"drop_add_residual_stochastic_depthr- )r3 r9 r> r@ r6 r= r7 �forwardY s& �� z
Block.forward)�__name__�25__module__� __qualname__r �GELU� LayerNormr26 r r/ �float�boolr �Moduler( r rD �
__classcell__r6 r6 r4 r7 r + s: � �-r r )r9 rA r2 r" c
C s� | j \}}}tt|d| �d�}tj|| jd�d |� }| | }||�} | �d�}27| �d�} || } tj|28d|| j| j d�| d�} | �29| �S )Nr ��devicer ��dtype��alpha) �shape�maxr/ �torch�randpermrO �flatten� index_add�torQ �view_as)
r9 rA r2 �b�n�d�sample_subset_size�brangeZx_subset�residual�x_flat�residual_scale_factor�x_plus_residualr6 r6 r7 rC u s 3031 rC c C sJ | j \}}}tt|d| �d�}tj|| jd�d |� }|| }||fS )Nr rN )rT rU r/ rV rW rO )r9 r2 r\ r] r^ r_ r` rc r6 r6 r7 �get_branges_scales� s32 re c C s\ |d u r<| � d�}|� d�}tj|d||j| jd�|d�}n t| ||j| jd�||d�}|S )Nr r rP rR )�scalingrS )rX rV rY rZ rQ r )r9 r` ra rc �scaling_vectorrb rd r6 r6 r7 � add_residual� s 3334 �rh �attn_bias_cachec C s� |durdd� |D �n dd� | D �}t dd� t|| �D ��}|t�� vr�g }t|| �D ]&\}}t|�D ]}|�|jd � qfqVtj� |�}||_35|t|< |dur�t dd� | D �|�� dd | d36 jd �} n t d d� | D ��}37t
j|38dd �} t| | fS )
zc39 this will perform the index select, cat the tensors, and provide the attn_bias from cache40 Nc S s g | ]}|j d �qS �r �rT )�.0r\ r6 r6 r7 �41<listcomp>� � z)get_attn_bias_and_cat.<locals>.<listcomp>c S s g | ]}|j d �qS rj rk �rl r9 r6 r6 r7 rm � rn c s s | ]\}}||j d fV qdS )r Nrk )rl r\ r9 r6 r6 r7 � <genexpr>� rn z(get_attn_bias_and_cat.<locals>.<genexpr>r c S s g | ]}|� d ��qS �r )rX ro r6 r6 r7 rm � rn �����r c s s* | ]"}|� d dg|jdd� ��V qdS )r rr � N)�reshaperT ro r6 r6 r7 rp � rn )r )�tuple�zipri �keys�range�appendrT r �BlockDiagonalMask� from_seqlens� _batch_sizesr �viewrV �cat) �x_list�branges� batch_sizesZ42all_shapes�seqlensr\ r9 �_� attn_biasZ cat_tensors� tensors_bs1r6 r6 r7 �get_attn_bias_and_cat� s $ *r� )r rA r2 r" c s� � fdd�| D �}dd� |D �}dd� |D �}t | |�\}}|�|||d��} g }43t| || |�D ](\} } }
}|44�t| | |
||��| �� q`|45S )Nc s g | ]}t |� d ��qS )�r2 )re ro r� r6 r7 rm � rn z;drop_add_residual_stochastic_depth_list.<locals>.<listcomp>c S s g | ] }|d �qS rj r6 �rl �sr6 r6 r7 rm � rn c S s g | ] }|d �qS rq r6 r� r6 r6 r7 rm � rn �r� )r� �splitrv ry rh r[ )r rA r2 rg Zbranges_scalesr� Zresidual_scale_factorsr� Zx_catZ
residual_list�outputsr9 r` ra rc r6 r� r7 �'drop_add_residual_stochastic_depth_list� s r� c s4 e Zd Zee ee d�dd�Z� fdd�Z� ZS )�NestedTensorBlock)r r" c s� t � jt�sJ �� jr�� jdkr�d ttd�� fdd�
}d
ttd�� fdd�
}t||� jt � jt�rj� jj ndd�}t||� jt � jt�r�� j46j ndd�}|S dttd�� fd d�
}dttd�� fd47d�
}t |�\}}||||d � }|||� }|� |�S dS )zL48 x_list contains a list of tensors to nest together and run49 r Nr8 c s � j � �| �|d�S �Nr� )r* r) �r9 r� r= r6 r7 r> � s z<NestedTensorBlock.forward_nested.<locals>.attn_residual_funcc s � � � �| ��S r: )r0 r. r� r= r6 r7 r@ � s z;NestedTensorBlock.forward_nested.<locals>.ffn_residual_func)rA r2 rg c s � � � j� �| �|d��S r� r; r� r= r6 r7 r> � s c s � � � �� �| ���S r: r? r� r= r6 r7 r@ � s r� )N)N)N)N)
�50isinstancer* r rB r2 r r� r, r
�gammar1 r� r� )r3 r r> r@ r� r9 r6 r= r7 �forward_nested� s. �� z NestedTensorBlock.forward_nestedc s> t |t�rt� �|�S t |t�r6ts,td��| �|�S t�d S )Nz-xFormers is required for using nested tensors)r� r r' rD �list�XFORMERS_AVAILABLE�AssertionErrorr� )r3 Z x_or_x_listr4 r6 r7 rD � s 51 5253zNestedTensorBlock.forward)rE rF rG r r r� rD rM r6 r6 r4 r7 r� � s (r� )r )r )N)N)r N),�logging�os�typingr r r r r �warningsrV r r � attentionr54 r r r Z layer_scaler
r0 r � getLogger�logger�environ�get�XFORMERS_ENABLED� xformers.opsr r r r� �warn� ImportErrorrL r rJ rC re rh ri �__annotations__r� r� r� r6 r6 r6 r7 �<module>55 sT 56 57 58 M � � 5960 61 ��