Apppublic
Aluode/PerceptionLabPortable

sourceHugging Faceupdated 9mo agoView on Hugging Face
0likes
rootpython_embedLibsite-packagestransformers__pycache__tf_utils.cpython-310.pyc
tf_utils.cpython-310.pyc108 linesDownload Raw Back to __pycache__
1o

2.�Yi~,�	@sddlmZmZddlZddlZddlmZddl	m3Z4ddlmZe�
e�Zdeejejfdeefd	d5�Zd+dejdeed
eedejfdd�Zd,dd�Z	d-deefdd�Zd.dd�Zdejdejfdd�Zd/dejded eddfd!d"�Zd#d$�Zd%d&�Zd'd(�Zd)d*�Z dS)0�)�Optional�UnionN�)�BatchFeature)�
BatchEncoding)�logging�tensor�returncsTt|tj�rt|j�St�|��|jt�d�kr�S|j��}�fdd�t|�D�S)z�6    Deal with dynamic shape in tensorflow cleanly.7 8    Args:9        tensor (`tf.Tensor` or `np.ndarray`): The tensor we want the shape of.10 11    Returns:12        `list[int]`: The shape of the tensor as a list.13    Ncs$g|]\}}|dur�|n|�qS�N�)�.0�i�s�Zdynamicr�E:\DocsHouse\542 percep lab latest\PerceptionLab\PerceptionLab_Portable\python_embed\Lib\site-packages\transformers/tf_utils.py�14<listcomp>0�$zshape_list.<locals>.<listcomp>)	�15isinstance�np�ndarray�list�shape�tfZTensorShapeZas_list�	enumerate)rZstaticrrr�16shape_lists17 181920r�logits�axis�namecCstjj|d||d�S)a�21    Stable wrapper that returns the same output as `tf.nn.softmax`, but that works reliably with XLA on CPU. It is22    meant as a workaround for the [following issue](https://github.com/tensorflow/tensorflow/issues/55682), and will be23    removed after it gets fixed. The arguments and outputs are the same as `tf.nn.softmax`, and relies on the fact that24    `softmax(x) = softmax(x + c)` (see https://ogunlao.github.io/2020/04/26/you_dont_really_know_softmax.html).25 26    Args:27        logits (`tf.Tensor`):28            Must be one of the following types: half, float32, float64.29        axis (`int`, *optional*):30            The dimension softmax would be performed on. The default is -1 which indicates the last dimension.31        name (`str`, *optional*):32            A name for the operation.33 34    Returns:35        `tf.Tensor`:36            A Tensor. Has the same type and shape as logits.37    g��&�.>�rrr)r�nn�softmaxrrrr�stable_softmax3sr!��h㈵��>�����c	Cs�|jjdks|jjdkst|t�std��tjj||gdd�\}}|dkr@dg|jj}t|�|||<t�	||�}t�	||�}tjj38||||||d�}|S)NrzOOnly 1D weight and bias tensors are supported for now, with only a single axis.T)ZaxesZkeepdimsr#)�offset�scaleZvariance_epsilon)r�rankr�int�NotImplementedErrorrrZmomentsr�reshapeZbatch_normalization)	Zinputs�weightZbias�epsilonr�meanZvariancerZoutputsrrr�functional_layernormKs""�r-�Fr%c	Cs�|dkrtd��|r|durtd��|r1tjt�|�dt�|�dftjd�}tjjj|dd�}|durP|jj	s=|jj39rPt�|dkt�d|j�t�d	|j��}t�
d40||�}|durit�t�|�d|j�d}||9}|duru||7}tj�|�}||S)
zDTF equivalent for torch's nn.functional.scaled_dot_product_attentionr.z�Dropout is not supported in this implementation - file an issue with Transformers and ping @Rocketknight1 if you need it for a port!Nz?You cannot specify an attn_mask and is_causal at the same time!�������dtyper)�kg@��z...qd, ...kd -> ...qkr#g�)�41ValueErrorrZonesrZint32Zexperimental�numpyZtrilr1�42is_integerZis_bool�where�castZeinsumrr )	�query�key�valueZ	attn_maskZ	dropout_pZ	is_causalr%rZprobsrrr�scaled_dot_product_attentionks&�(&r;cCs�|dkr43||jj7}|dkr||jj7}||kr|St�|�}tj�|||d��}tj|d|�|g||dd�gdd�}t�||�S)Nrr�r)rr&r�mathZreduce_prod�concatr))�inputZ	start_dimZend_dimZin_shapeZ
flattened_dimZ	out_shaperrr�flatten�s44*r@�encoder_attention_maskcCs�t|tj�st�|�}|jjdkr|dd�ddd�dd�f}|jjdkr1|dd�dddd�f}t�d|j�||jj}|S)z�45    Invert an attention mask (e.g., switches 0. and 1.).46 47    Args:48        encoder_attention_mask (`torch.Tensor`): An attention mask.49 50    Returns:51        `tf.Tensor`: The inverted attention mask.52    �N�r)	rr�TensorZconvert_to_tensorrr&r7r1�min)rAZencoder_extended_attention_maskrrr�invert_attention_mask�s53 54�rF�	input_ids�	embed_dim�tensor_namec55Cs>tjj|tj||jd�d|�dtj�|��d|�d�d�dS)a�56    `tf.gather`, on which TF embedding layers are based, won't check positive out of bound indices on GPU, returning57    zeros instead. This function adds a check against that dangerous silent behavior.58 59    Args:60        tensor (`tf.Tensor`): The tensor of indices to check.61        embed_dim (`int`): The embedding dimension.62        tensor_name (`str`, *optional*): The name of the tensor to use in the error message.63    r0zThe maximum value of z (z>) must be smaller than the embedding layer's input dimension (z9). The likely cause is some problem at tokenization time.)�messageN)rZ	debuggingZassert_lessr7r1r=Z64reduce_max)rrHrIrrr�check_embeddings_within_bounds�s65�66�rKc	s�d��fdd�|D�}|rtd��d|����t�|�}d}t�||�}t�fdd�|D��rD|d7}t�||�}t�fdd�|D��s/|dkr\t|�D]
\}}||jd	||f<qLd67S||j|<d68S)a�Saves attributes (data) of the specified name into the HDF5 group.69 70    This method deals with an inherent problem of HDF5 file which is not able to store data larger than71    HDF5_OBJECT_HEADER_LIMIT bytes.72 73    Args:74        group: A pointer to a HDF5 group.75        name: A name of the attributes to save.76        data: Attributes data to store.77 78    Raises:79      RuntimeError: If any single attribute is too large to be saved.80 81    Copied from Keras to Transformers to avoid versioning issues.82    i�csg|]83}t|��kr|�qSr)�len�r�x�ZHDF5_OBJECT_HEADER_LIMITrrr�sz1save_attributes_to_hdf5_group.<locals>.<listcomp>zSThe following attributes cannot be saved to HDF5 file because they are larger than z bytes: rc3s�|]}|j�kVqdSr84)�nbytesrMrOrr�	<genexpr>�s�z0save_attributes_to_hdf5_group.<locals>.<genexpr>�%s%dN)�RuntimeErrorrZasarrayZarray_split�anyr�attrs)	�groupr�dataZbad_attributesZdata_npyZ85num_chunksZchunked_data�chunk_idZ86chunk_datarrOr�save_attributes_to_hdf5_group�s,���87��rYcCs|||jvrdd�|j|D�}|Sg}d}d||f|jvr<|�dd�|jd||fD��|d7}d||f|jvs|S)a�Loads attributes of the specified name from the HDF5 group.88 89    This method deals with an inherent problem of HDF5 file which is not able to store data larger than90    HDF5_OBJECT_HEADER_LIMIT bytes.91 92    Args:93        group: A pointer to a HDF5 group.94        name: A name of the attributes to load.95 96    Returns:97        data: Attributes data.98 99    Copied from Keras to Transformers to avoid versioning issues.100    cS�$g|]}t|d�r|�d�n|�qS��decode�utf8��hasattrr\�r�nrrrrrz3load_attributes_from_hdf5_group.<locals>.<listcomp>rrRcSrZr[r^r`rrrr
rr)rU�extend)rVrrWrXrrr�load_attributes_from_hdf5_group�s101	���rccCsdd�}tj�||�S)zwExpands 1-dimensional `Tensor`s into 2-dimensional `Tensor`s.102    Copied from Keras to here to avoid versioning issues.cSs*t|tj�r|jjdkrtj|dd�S|S)Nrr#r<)rrrDrr&Zexpand_dims)�trrr�_expand_single_1d_tensorsz+expand_1d.<locals>._expand_single_1d_tensor)r�nestZ
map_structure)rWrerrr�	expand_1dsrgcOsh|rt|dttf�rt|�}t|d�|d<||fSd|vr0t|dttf�r0t|d�|d<||fS)NrrN)rrrr�dict)�args�kwargsrrr�convert_batch_encodings�rk)NN)r"r#)Nr.FN)rr#)rG)!�typingrrr4rZ103tensorflowrZfeature_extraction_utilsrZtokenization_utils_baser�utilsrZ104get_logger�__name__�loggerrDrrr'r�strr!r-�floatr;r@rFrKrYrcrgrkrrrr�<module>s*105"(106!�107�108/