TTXian/RemoteSensingChangeDetection-RSCD.HA2F
0
1a
2 8��g� � @ s� d dl Z d dlmZmZ d dlZd dlmZmZ d dlm m Z3 G dd� dej �Z e j
�d�du Zz2er�d dlmZ dZe�d � ne�d4� e�W n$ ey� e Zd Ze�d � Y n0 G d
d� de�ZdS )� N)�Callable�Optional)�Tensor�nnc sR e Zd Zd eee ee edejf ee dd�� fdd�
Z5e e d�d d6�Z � Z
S ) � SwiGLUFFNN� T.�� in_features�hidden_features� out_features� act_layer�drop�bias�returnc sF t � �� |p|}|p|}tj|d| |d�| _tj|||d�| _d S )N� )r )�super�__init__r �Linear�w12�w3��selfr r7 r r r
r �� __class__� �GC:\Users\82607\Desktop\code\ChangeViT-master\model\layers\swiglu_ffn.pyr s8 9zSwiGLUFFN.__init__)�xr c C s4 | � |�}|jddd�\}}t�|�| }| �|�S )Nr �����)�dim)r �chunk�F�silur )r r �x12�x1�x2�hiddenr r r �forward s 10zSwiGLUFFN.forward)NNNr T)�__name__�11__module__� __qualname__�intr r r �Module�float�boolr r r& �
__classcell__r r r r r s � �r ZXFORMERS_DISABLED)�SwiGLUTz xFormers is available (SwiGLU)z xFormers is disabled (SwiGLU)Fz"xFormers is not available (SwiGLU)c sB e Zd Zdeee ee edejf ee dd�� fdd�
Z12� Z S ) �SwiGLUFFNFusedNr T.r c sD |p|}|p|}t |d d �d d d }t� j||||d� d S )Nr � � � )r r13 r r )r* r r r r r r r 7 s �zSwiGLUFFNFused.__init__)NNNr T) r' r( r) r* r r r r+ r, r- r r. r r r r r0 6 s � �r0 )�os�typingr r �warnings�torchr r �torch.nn.functional�14functionalr r+ r �environ�getZXFORMERS_ENABLEDZ xformers.opsr/ ZXFORMERS_AVAILABLE�warn� ImportErrorr0 r r r r �<module> s$ 15