Apppublic
declare-lab/tango2

sourceHugging Faceupdated 2y agoView on Hugging Face
92likes
rootdiffuserssrcdiffusers__pycache__training_utils.cpython-39.pyc
training_utils.cpython-39.pyc115 linesDownload Raw Back to __pycache__
1a

2��'dw4�@s~ddlZddlZddlZddlmZmZmZmZmZddl	Z3ddlZddlm
Z
ed�dd�Zed�dd	�ZGd4d�d�ZdS)�N)�Any�Dict�Iterable�Optional�Union�)�	deprecate��seedcCs>t|�dtjd<dtjd<t�d�dtjj_dtjj_dS)z�5    Helper function for reproducible behavior during distributed training. See6    - https://pytorch.org/docs/stable/notes/randomness.html for pytorch7    �1ZCUDA_LAUNCH_BLOCKINGz:16:8ZCUBLAS_WORKSPACE_CONFIGTFN)	�set_seed�os�environ�torch�use_deterministic_algorithms�backends�cudnn�
deterministic�	benchmarkr	�r�M/home/deep/Projects/audio_diffusion/diffusers/src/diffusers/training_utils.py�enable_full_determinisms891011rcCs0t�|�tj�|�t�|�tj�|�dS)z�12    Args:13    Helper function for reproducible behavior to set the seed in `random`, `numpy`, `torch`.14        seed (`int`): The seed to set.15    N)�randomr16�npr�manual_seed�cuda�manual_seed_allr	rrrr s1718rc@seZdZdZd%eejjeee	e19eee	feee	fee
eee
fd	�	d20d�Zedd�d
d��Zdd�Ze	ed�dd�Ze��eejjd�dd��Zeejjdd�dd�Zd&dd�dd�Zed�dd�Zeejjdd�dd�Zeejjdd�d d!�Zedd"�d#d$�ZdS)'�EMAModelz621    Exponential Moving Average of models weights22    ���H.��?�rF��?�UUUUUU�?N)	�23parameters�decay�	min_decay�update_after_step�use_ema_warmup�	inv_gamma�power�	model_cls�model_configc24Kst|tjj�r.d}tdd|dd�|��}d}|25�dd�durZd	}tdd|dd�|26d}|27�d28d�dur�d}td29d|dd�|30d31}t|�}dd
�|D�|_|32�dd�dur�d}tdd|dd�|j	|33dd�d|_34||_||_||_
||_||_||_d|_d|_||_|	|_dS)a35        Args:36            parameters (Iterable[torch.nn.Parameter]): The parameters to track.37            decay (float): The decay factor for the exponential moving average.38            min_decay (float): The minimum decay factor for the exponential moving average.39            update_after_step (int): The number of steps to wait before starting to update the EMA weights.40            use_ema_warmup (bool): Whether to use EMA warmup.41            inv_gamma (float):42                Inverse multiplicative factor of EMA warmup. Default: 1. Only used if `use_ema_warmup` is True.43            power (float): Exponential factor of EMA warmup. Default: 2/3. Only used if `use_ema_warmup` is True.44            device (Optional[Union[str, torch.device]]): The device to store the EMA weights on. If None, the EMA45                        weights will be stored on CPU.46 47        @crowsonkb's notes on EMA Warmup:48            If gamma=1 and power=1, implements a simple average. gamma=1, power=2/3 are good values for models you plan49            to train for a million or more steps (reaches decay factor 0.999 at 31.6K steps, 0.9999 at 1M steps),50            gamma=1, power=3/4 for models you plan to train for less (reaches decay factor 0.999 at 10K steps, 0.999951            at 215.4k steps).52        zzPassing a `torch.nn.Module` to `ExponentialMovingAverage` is deprecated. Please pass the parameters of the module instead.z9passing a `torch.nn.Module` to `ExponentialMovingAverage`�1.0.0F��
standard_warnT�	max_valueNzCThe `max_value` argument is deprecated. Please use `decay` instead.�	min_valuezGThe `min_value` argument is deprecated. Please use `min_decay` instead.cSsg|]}|�����qSr)�clone�detach��.0�prrr�53<listcomp>o�z%EMAModel.__init__.<locals>.<listcomp>�devicez=The `device` argument is deprecated. Please use `to` instead.�r7r)�54isinstancer�nn�Modulerr"�get�list�
shadow_params�to�temp_stored_paramsr#r$r%r&r'r(�optimization_step�cur_decay_valuer)r*)�selfr"r#r$r%r&r'r(r)r*�kwargs�deprecation_messagerrr�__init__3sH!��zEMAModel.__init__)�returncCs>|j|dd�\}}|�|�}||��||jd�}|�|�|S)NT)�return_unused_kwargs)r)r*)�load_config�from_pretrainedr"�config�load_state_dict)�cls�pathr)�_Z55ema_kwargs�modelZ	ema_modelrrrrJ�s565758zEMAModel.from_pretrainedcCsr|jdurtd��|jdur$td��|j�|j�}|��}|�dd�|jfi|��|�|���|�	|�dS)NzJ`save_pretrained` can only be used if `model_cls` was defined at __init__.zM`save_pretrained` can only be used if `model_config` was defined at __init__.r>)59r)�60ValueErrorr*�from_config�61state_dict�pop�register_to_config�copy_tor"�save_pretrained)rCrNrPrSrrrrW�s6263zEMAModel.save_pretrained)rArGcCsntd||jd�}|dkr dS|jrBdd||j|j}nd|d|}t||j�}t||j�}|S)zN64        Compute the decay factor for the exponential moving average.65        rrr�66)�maxr%r&r'r(�minr#r$)rCrA�steprBrrr�	get_decay�szEMAModel.get_decay)r"cCs�t|tjj�r*d}tdd|dd�|��}t|�}|jd7_|�|j�}||_	d|}t67|j|�D],\}}|jr�|�
|||�qf|�|�qfdS)NzPassing a `torch.nn.Module` to `ExponentialMovingAverage.step` is deprecated. Please pass the parameters of the module instead.z>passing a `torch.nn.Module` to `ExponentialMovingAverage.step`r+Fr,r)r9rr:r;rr"r=rAr\rB�zipr>�
requires_grad�sub_�copy_)rCr"rEr#Zone_minus_decay�s_param�paramrrrr[�s&��z
EMAModel.step)r"rGcCs8t|�}t|j|�D]\}}|j�|�|j�j�qdS)aa68        Copy current averaged parameters into given collection of parameters.69 70        Args:71            parameters: Iterable of `torch.nn.Parameter`; the parameters to be72                updated with the stored moving averages. If `None`, the parameters with which this73                `ExponentialMovingAverage` was initialized will be used.74        N)r=r]r>�datar`r?r7)rCr"rarbrrrrV�s	zEMAModel.copy_tocs��fdd�|jD�|_dS)z�Move internal buffers of the ExponentialMovingAverage to `device`.75 76        Args:77            device: like `device` argument to `torch.Tensor.to`78        cs.g|]&}|��r|j��d�n79|j�d��qS)�r7�dtyper8)�is_floating_pointr?r2rdrrr5�s�zEMAModel.to.<locals>.<listcomp>N)r>)rCr7rerrdrr?�s�zEMAModel.toc	Cs&|j|j|j|j|j|j|j|jd�S)z�80        Returns the state of the ExponentialMovingAverage as a dict. This method is used by accelerate during81        checkpointing to save the ema state dict.82        �r#r$rAr%r&r'r(r>rg)rCrrrrS�s	�zEMAModel.state_dictcCsdd�|D�|_dS)z�83        Args:84        Save the current parameters for restoring later.85            parameters: Iterable of `torch.nn.Parameter`; the parameters to be86                temporarily stored.87        cSsg|]}|�������qSr)r1�cpur0)r3rbrrrr5r6z"EMAModel.store.<locals>.<listcomp>N)r@)rCr"rrr�store�szEMAModel.storecCs@|jdurtd��t|j|�D]\}}|j�|j�qd|_dS)aF88        Args:89        Restore the parameters stored with the `store` method. Useful to validate the model with EMA parameters without:90        affecting the original optimization process. Store the parameters before the `copy_to()` method. After91        validation (or model saving), use this to restore the former parameters.92            parameters: Iterable of `torch.nn.Parameter`; the parameters to be93                updated with the stored parameters. If `None`, the parameters with which this94                `ExponentialMovingAverage` was initialized will be used.95        NzGThis ExponentialMovingAverage has no `store()`ed weights to `restore()`)r@�RuntimeErrorr]rcr`)rCr"Zc_paramrbrrr�restores9697 98zEMAModel.restore)rSrGcCslt�|�}|�d|j�|_|jdks.|jdkr6td��|�d|j�|_t|jt�sZtd��|�d|j�|_t|jt	�s~td��|�d	|j99�|_100t|j101t	�s�td102��|�d|j�|_t|jt�s�td��|�d
|j
�|_
t|j
tt	f�s�td��|�d|j�|_t|jtt	f��std��|�dd�}|du�rh||_t|jt��sJtd��tdd�|jD���shtd��dS)a103        Args:104        Loads the ExponentialMovingAverage state. This method is used by accelerate during checkpointing to save the105        ema state dict.106            state_dict (dict): EMA state. Should be an object returned107                from a call to :meth:`state_dict`.108        r#rr zDecay must be between 0 and 1r$zInvalid min_decayrAzInvalid optimization_stepr%zInvalid update_after_stepr&zInvalid use_ema_warmupr'zInvalid inv_gammar(z
Invalid powerr>Nzshadow_params must be a listcss|]}t|tj�VqdS)N)r9r�Tensorr2rrr�	<genexpr>Ar6z+EMAModel.load_state_dict.<locals>.<genexpr>z!shadow_params must all be Tensors)�copy�deepcopyr<r#rQr$r9�floatrA�intr%r&�boolr'r(r>r=�all)rCrSr>rrrrLs:	109110zEMAModel.load_state_dict)rrrFr r!NN)NN)�__name__�111__module__�__qualname__�__doc__rrr:�	Parameterrprqrrrrrr�strrF�classmethodrJrWr\�no_gradr[rVr?�dictrSrirkrLrrrrr.s@�112113114115�Q	
	r)rnr
r�typingrrrrr�numpyrr�utilsrrqrrrrrrr�<module>s