Aluode/PerceptionLabPortable
0
1o
2 .�Yiy �
@ s3 d dl Z d dlZd dlmZ zd dlZW n eefy/ d dlZeej �j4dkr-e d��Y nw dd� Z dd� Z
d d5� Zd d � Zd
d� Zdd� Zddd�Zeejj�ed�krbdd� ZejjZeZne Ze
Zeeeeeeeejjejjejjejjejj d� Z dd� Z dS )� N)�parse� z�Your currently installed version of Keras is Keras 3, but this is not yet supported in Transformers. Please install the backwards-compatible tf-keras package with `pip install tf-keras`.c 6 C s: t �| �} ddt j�| t �t �d�| j� � }| | S )a~ 7 Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when8 initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results):9 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) Also see10 https://huggingface.co/papers/1606.0841511 � �?� �?� @)�tf�convert_to_tensor�math�erf�cast�sqrt�dtype)�x�cdf� r ��E:\DocsHouse\542 percep lab latest\PerceptionLab\PerceptionLab_Portable\python_embed\Lib\site-packages\transformers/activations_tf.py�_gelu" s 12(r c C s` t �| �} t �tj| j�}t �d| j�}ddt �t �d| �| |t �| d� � }| | S )z�13 Gaussian Error Linear Unit. This is a smoother version of the GELU. Original paper: https://huggingface.co/papers/1606.084114 15 Args:16 x: float Tensor to perform activation17 18 Returns:19 `x` with the GELU activation applied.20 ��Hm��?r r r � ) r r r r �pir
�tanhr �pow)r r �coeffr r r r � _gelu_new/ s21 22 230r c C s t �| �} | t �t j�| �� S )N)r r r r Zsoftplus�r r r r �mishA s 24r c C sP t �| �} t �d| j�}t �d| j�}d| dt �| | d|| | � S )Nr g���3E��?r r )r r r r
r )r Zcoeff1Zcoeff2r r r � gelu_fastG s 25*r c C s, t �| �} t �d| j�}| t j�|| � S )Ng� Zd;�?)r r r r
r �sigmoid)r r r r r �26quick_geluO s 27r c C s t �t| �dd�S )a� 28 Clip the range of possible GeLU outputs between [-10, 10]. This is especially useful for quantization purpose, as29 it allows mapping 2 negatives values in the GeLU spectrum. For more information on this trick, please refer to30 https://huggingface.co/papers/2004.0960231 32 Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when33 initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results):34 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) Also see35 https://huggingface.co/papers/1606.08415 :param x: :return:36 i�����37 )r Z
clip_by_valuer r r r r �gelu_10U s r �����c C s$ t j| d|d�\}}|t j�|� S )a� 38 Gated Linear Unit. Implementation as defined in the original paper (see https://huggingface.co/papers/1612.08083), where39 the input `x` is split in two halves across a dimension (`axis`), A and B, returning A * sigmoid(B).40 41 Args:42 `x`: float Tensor to perform activation43 `axis`: dimension across which `x` be split in half44 45 Returns:46 `x` with the GLU activation applied (with its size halved across the dimension `axis`).47 r )�axis)r �splitr r )r r" �a�br r r �gluc s r& z2.4c C s t jj| dd�S )NT)Z approximate)�keras� activations�gelur r r r �approximate_gelu_wrapu s r* ) r) r r �gelu_newr& r r �relur Zsilu�swishr c C s, | t v rt | S td| � dtt �� �� ���)Nz function z not found in ACT2FN mapping )�ACT2FN�KeyError�list�keys)Zactivation_stringr r r �get_tf_activation� s r2 )r! ) r Z48tensorflowr Zpackaging.versionr Ztf_kerasr' �ModuleNotFoundError� ImportError� __version__�major�49ValueErrorr r r r r r r& �version�VERSIONr* r( r) r+ r, r r- r r. r2 r r r r �<module> sP ���
50�