Apppublic
xdecoder/Instruct-X-Decoder

sourceHugging Faceafl-3.0updated 3y agoView on Hugging Face
163likes
util.cpython-38.pyc149 linesDownload Raw Back to __pycache__
1U

2E$�c�'�@s$ddlZddlZddlZddlZddlmZddlZddlm	Z	dd�Z3d4dd�Zd5dd�Zd6dd�Z
d7dd�Zd8dd�Zdd�Zdd�ZGdd�dejj�Zd9dd�Zdd�Zd d!�Zd"d#�Zd$d%�ZGd&d'�d'ej�ZGd(d)�d)ej�Zd*d+�Zd,d-�Zd.d/�Z Gd0d1�d1ej�Z!d:d2d3�Z"dS);�N)�repeatcCsDd|kr(|dkrdS|dkr dStd��t|d�f|�dt���S)N�targetZ__is_first_stage__Z__is_unconditional__z%Expected key `target` to instantiate.�params)�KeyError�get_obj_from_str�get�dict)�config�r4�#/data/arXiv/demo/Demo/utils/util.py�instantiate_from_configsrFcCs<|�dd�\}}|r(t�|�}t�|�ttj|dd�|�S)N�.�)�package)�rsplit�	importlib�
import_module�reload�getattr)�stringr�module�clsZ5module_impr6r7rrs8910r�-C��6?�{�G�z�?�����Mb�?cCs|dkr*tj|d|d|tjd�d}n�|dkr�tj|dtjd�||}|d|tjd}t�|��d�}||d}d|dd�|dd�}tj|dd	d11�}nP|dkr�tj|||tjd�}n2|dkr�tj|||tjd�d}nt	d
|�d���|�12�S)N�linearg�?)�dtype��cosinerr������+�����?)�a_min�a_maxZsqrt_linear�sqrtz13schedule 'z14' unknown.)�torch�linspace�float64�arange�np�pi�cos�pow�clip�15ValueError�numpy)�scheduleZ16n_timestepZlinear_startZ17linear_endZcosine_s�betas�	timesteps�alphasr18r19r�make_beta_schedule%s"��r3TcCs�|dkr(||}t�ttd||���}n<|dkrTt�dt�|d�|�d�t�}ntd|�d���|d}|r~t	d	|���|S)20N�uniformr�quadg�������?rz/There is no ddim discretization method called "�"rz%Selected timesteps for ddim sampler: )21r(�asarray�list�ranger%r#�astype�int�NotImplementedError�print)�ddim_discr_method�num_ddim_timesteps�num_ddpm_timesteps�verbose�c�ddim_timestepsZ	steps_outr22r23r�make_ddim_timesteps>s$rDcCs�||}t�|dg||dd����}|t�d|d|d||�}|r~td|�d|���td|�d|���|||fS)Nrrrz'Selected alphas for ddim sampler: a_t: z; a_(t-1): z&For the chosen value of eta, which is zB, this results in the following sigma_t schedule for ddim sampler )r(r7�tolistr#r=)�	alphacumsrC�etarAr2�alphas_prev�sigmasr24r25r�make_ddim_sampling_parametersOs$&rJr cCsPg}t|�D]8}||}|d|}|�td||�||�|��qt�|�S)a#26    Create a beta schedule that discretizes the given alpha_t_bar function,27    which defines the cumulative product of (1-beta) over time from t = [0,1].28    :param num_diffusion_timesteps: the number of betas to produce.29    :param alpha_bar: a lambda that takes an argument t from 0 to 1 and30                      produces the cumulative product of (1-beta) up to that31                      part of the diffusion process.32    :param max_beta: the maximum beta to use; use values lower than 1 to33                     prevent singularities.34    r)r9�append�minr(�array)Znum_diffusion_timestepsZ	alpha_barZmax_betar0�i�t1�t2r35r36r�betas_for_alpha_bar]s"rQcCs2|j^}}|�d|�}|j|fdt|�d��S)Nr�rr)�shape�gather�reshape�len)�a�t�x_shape�b�_�outr37r38r�extract_into_tensorps39r]cCs6|r*t|�t|�}tj|t|�f|��S||�SdS)a�40    Evaluate a function without caching intermediate activations, allowing for41    reduced memory at the expense of extra compute in the backward pass.42    :param func: the function to evaluate.43    :param inputs: the argument sequence to pass to `func`.44    :param params: a sequence of parameters `func` depends on but does not45                   explicitly take as arguments.46    :param flag: if False, disable gradient checkpointing.47    N)�tuple�CheckpointFunction�applyrV)�func�inputsr�flag�argsr48r49r�50checkpointvs51rec@s$eZdZedd��Zedd��ZdS)r_c	GsN||_t|d|��|_t||d��|_t���|j|j�}W5QRX|S�N)�run_functionr8�
input_tensors�input_paramsr$�no_grad)�ctxrg�lengthrd�output_tensorsr52r53r�forward�s54zCheckpointFunction.forwardc	Gsndd�|jD�|_t��� dd�|jD�}|j|�}W5QRXtjj||j|j|dd�}|`|`~d|S)NcSsg|]}|���d��qS)T)�detach�requires_grad_��.0�xr55r56r�57<listcomp>�sz/CheckpointFunction.backward.<locals>.<listcomp>cSsg|]}|�|��qSr58)�view_asrqr59r60rrt�sT)�allow_unused)NN)rhr$�enable_gradrg�autograd�gradri)rkZoutput_gradsZshallow_copiesrmZinput_gradsr61r62r�backward�s6364�zCheckpointFunction.backwardN)�__name__�65__module__�__qualname__�staticmethodrnrzr66r67r68rr_�s69	r_�'cCs�|s�|d}t�t�|�tjd|tjd�|�j|jd�}|dd�df��|d}tj	t�70|�t�|�gdd�}|dr�tj	|t�|dd�dd�f�gdd�}nt
|d	|d71�}|S)aX72    Create sinusoidal timestep embeddings.73    :param timesteps: a 1-D Tensor of N indices, one per batch element.74                      These may be fractional.75    :param dim: the dimension of the output.76    :param max_period: controls the minimum frequency of the embeddings.77    :return: an [N x dim] Tensor of positional embeddings.78    rr)�start�endr��deviceNr��dimrzb -> b d)�d)r$�exp�math�logr'�float32�tor��float�catr*�sin�79zeros_liker)r1r�Z80max_periodZrepeat_only�half�freqsrd�	embeddingr81r82r�timestep_embedding�s	 ��*r�cCs|��D]}|����q|S)z<83    Zero out the parameters of a module and return it.84    )�85parametersro�zero_)r�pr86r87r�zero_module�sr�cCs |��D]}|���|�q|S)z988    Scale the parameters of a module and return it.89    )r�ro�mul_)r�scaler�r90r91r�scale_module�sr�cCs|jttdt|j���d�S)z692    Take the mean over all non-batch dimensions.93    rr�)�meanr8r9rVrS)�tensorr94r95r�	mean_flat�sr�cCs96td|�S)z�97    Make a standard normalization layer.98    :param channels: number of input channels.99    :return: an nn.Module for normalization.100    � )�GroupNorm32)�channelsr101r102r�
normalization�sr�c@seZdZdd�ZdS)�SiLUcCs|t�|�Srf)r$�sigmoid��selfrsr103r104rrn�szSiLU.forwardN)r{r|r}rnr105r106r107rr��sr�cseZdZ�fdd�Z�ZS)r�cst��|����|j�Srf)�superrnr��typerr���	__class__r108rrn�szGroupNorm32.forward)r{r|r}rn�
__classcell__r109r110r�rr��sr�cOsN|dkrtj||�S|dkr(tj||�S|dkr<tj||�Std|����dS)z4111    Create a 1D, 2D, or 3D convolution module.112    rr��unsupported dimensions: N)�nn�Conv1d�Conv2d�Conv3dr-��dimsrd�kwargsr113r114r�conv_nd�sr�cOstj||�S)z!115    Create a linear module.116    )r��Linear)rdr�r117r118rr�srcOsN|dkrtj||�S|dkr(tj||�S|dkr<tj||�Std|����dS)z8119    Create a 1D, 2D, or 3D average pooling module.120    rrr�r�N)r��	AvgPool1d�	AvgPool2d�	AvgPool3dr-r�r121r122r�avg_pool_nd�sr�cs$eZdZ�fdd�Zdd�Z�ZS)�HybridConditionercs"t���t|�|_t|�|_dSrf)r��__init__r�concat_conditioner�crossattn_conditioner)r�Zc_concat_configZc_crossattn_configr�r123rr�
s124125zHybridConditioner.__init__cCs"|�|�}|�|�}|g|gd�S)N)�c_concat�c_crossattn)r�r�)r�r�r�r126r127rrns128129zHybridConditioner.forward)r{r|r}r�rnr�r130r131r�rr�sr�cs,��fdd�}��fdd�}|r&|�S|�S)Ncs6tjd�dd���d�j�dfdt��d��S)Nrr�rrR)r)r$�randnrrVr132�r�rSr133r�<lambda>�znoise_like.<locals>.<lambda>cstj��d�S)Nr�)r$r�r134r�r135rr�r�r136)rSr�r�repeat_noise�noiser137r�r�138noise_likesr�)F)rrr)T)T)r )rF)F)#r�osr�r$�torch.nnr�r.r(�einopsrrrr3rDrJrQr]rerx�Functionr_r�r�r�r�r��Moduler��	GroupNormr�r�rr�r�r�r139r140r141r�<module>	s6142 143144145146147 148		149