Apppublic
declare-lab/tango2

sourceHugging Faceupdated 2y agoView on Hugging Face
92likes
rootdiffuserssrcdiffusers__pycache__optimization.cpython-39.pyc
optimization.cpython-39.pyc156 linesDownload Raw Back to __pycache__
1a

2��'d�0�@s(dZddlZddlmZddlmZmZddlmZddl	m3Z4ddlmZe�
e�ZGd	d5�d6e�Zd!eed�d
d�Zd"eeed�dd�Zd#dd�Zd$eeeeed�dd�Zd%eeeeed�dd�Zd&dd�ZejeejeejeejeejeejeiZd'ee efeeeeeeeed�dd �Z!dS)(z*PyTorch optimization for diffusion models.�N)�Enum)�Optional�Union)�	Optimizer��LambdaLR�)�loggingc@s$eZdZdZdZdZdZdZdZdS)�
SchedulerType�linearZcosineZcosine_with_restarts�7polynomial�constantZconstant_with_warmupN)	�__name__�8__module__�__qualname__�LINEAR�COSINE�COSINE_WITH_RESTARTS�9POLYNOMIAL�CONSTANT�CONSTANT_WITH_WARMUP�rr�K/home/deep/Projects/audio_diffusion/diffusers/src/diffusers/optimization.pyr10sr11�������	optimizer�12last_epochcCst|dd�|d�S)a�13    Create a schedule with a constant learning rate, using the learning rate set in optimizer.14 15    Args:16        optimizer ([`~torch.optim.Optimizer`]):17            The optimizer for which to schedule the learning rate.18        last_epoch (`int`, *optional*, defaults to -1):19            The index of the last epoch when resuming training.20 21    Return:22        `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule.23    cSsdS�Nrr)�_rrr�<lambda>4�z'get_constant_schedule.<locals>.<lambda>�rrrrrr�get_constant_schedule's
r")r�num_warmup_stepsrcs td��fdd�}t|||d�S)ad24    Create a schedule with a constant learning rate preceded by a warmup period during which the learning rate25    increases linearly between 0 and the initial lr set in the optimizer.26 27    Args:28        optimizer ([`~torch.optim.Optimizer`]):29            The optimizer for which to schedule the learning rate.30        num_warmup_steps (`int`):31            The number of steps for the warmup phase.32        last_epoch (`int`, *optional*, defaults to -1):33            The index of the last epoch when resuming training.34 35    Return:36        `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule.37    ��current_stepcs"|�krt|�ttd���SdS)N��?��float�maxr$�r#rr�	lr_lambdaHsz4get_constant_schedule_with_warmup.<locals>.lr_lambdar!��intr)rr#rr+rr*r�!get_constant_schedule_with_warmup7sr.cs td���fdd�}t|||�S)a�38    Create a schedule with a learning rate that decreases linearly from the initial lr set in the optimizer to 0, after39    a warmup period during which it increases linearly from 0 to the initial lr set in the optimizer.40 41    Args:42        optimizer ([`~torch.optim.Optimizer`]):43            The optimizer for which to schedule the learning rate.44        num_warmup_steps (`int`):45            The number of steps for the warmup phase.46        num_training_steps (`int`):47            The total number of training steps.48        last_epoch (`int`, *optional*, defaults to -1):49            The index of the last epoch when resuming training.50 51    Return:52        `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule.53    r$csB|�krt|�ttd���Stdt�|�ttd�����S)Nr�r'r$��num_training_stepsr#rrr+cs54�z2get_linear_schedule_with_warmup.<locals>.lr_lambdar,)rr#r1rr+rr0r�get_linear_schedule_with_warmupPsr2��?)rr#r1�55num_cyclesrcs���fdd�}t|||�S)a�56    Create a schedule with a learning rate that decreases following the values of the cosine function between the57    initial lr set in the optimizer to 0, after a warmup period during which it increases linearly between 0 and the58    initial lr set in the optimizer.59 60    Args:61        optimizer ([`~torch.optim.Optimizer`]):62            The optimizer for which to schedule the learning rate.63        num_warmup_steps (`int`):64            The number of steps for the warmup phase.65        num_training_steps (`int`):66            The total number of training steps.67        num_periods (`float`, *optional*, defaults to 0.5):68            The number of periods of the cosine function in a schedule (the default is to just decrease from the max69            value to 0 following a half-cosine).70        last_epoch (`int`, *optional*, defaults to -1):71            The index of the last epoch when resuming training.72 73    Return:74        `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule.75    c	sf|�krt|�ttd���St|��ttd����}tdddt�tjt��d|��S)Nrr/r3r&g@�r(r)�math�cos�pi�r%�progress�r4r1r#rrr+�sz2get_cosine_schedule_with_warmup.<locals>.lr_lambdar�rr#r1r4rr+rr;r�get_cosine_schedule_with_warmupmsr=cs���fdd�}t|||�S)a�76    Create a schedule with a learning rate that decreases following the values of the cosine function between the77    initial lr set in the optimizer to 0, with several hard restarts, after a warmup period during which it increases78    linearly between 0 and the initial lr set in the optimizer.79 80    Args:81        optimizer ([`~torch.optim.Optimizer`]):82            The optimizer for which to schedule the learning rate.83        num_warmup_steps (`int`):84            The number of steps for the warmup phase.85        num_training_steps (`int`):86            The total number of training steps.87        num_cycles (`int`, *optional*, defaults to 1):88            The number of hard restarts to use.89        last_epoch (`int`, *optional*, defaults to -1):90            The index of the last epoch when resuming training.91 92    Return:93        `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule.94    c	sr|�krt|�ttd���St|��ttd����}|dkrHdStdddt�tjt��|d��S)Nrr&r/r3r5r9r;rrr+�szEget_cosine_with_hard_restarts_schedule_with_warmup.<locals>.lr_lambdarr<rr;r�2get_cosine_with_hard_restarts_schedule_with_warmup�sr>�H�����z>r&csN|jd���ks(td��d��d���td������fdd�}t|||�S)a�95    Create a schedule with a learning rate that decreases as a polynomial decay from the initial lr set in the96    optimizer to end lr defined by *lr_end*, after a warmup period during which it increases linearly from 0 to the97    initial lr set in the optimizer.98 99    Args:100        optimizer ([`~torch.optim.Optimizer`]):101            The optimizer for which to schedule the learning rate.102        num_warmup_steps (`int`):103            The number of steps for the warmup phase.104        num_training_steps (`int`):105            The total number of training steps.106        lr_end (`float`, *optional*, defaults to 1e-7):107            The end LR.108        power (`float`, *optional*, defaults to 1.0):109            Power factor.110        last_epoch (`int`, *optional*, defaults to -1):111            The index of the last epoch when resuming training.112 113    Note: *power* defaults to 1.0 as in the fairseq implementation, which in turn is based on the original BERT114    implementation at115    https://github.com/google-research/bert/blob/f39e881b169b9d53bea03d2d341b31707a6c052b/optimization.py#L37116 117    Return:118        `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule.119 120    �lrzlr_end (z&) must be be smaller than initial lr (�)r$csj|�krt|�ttd���S|�kr.��S��}��}d|�|}||��}|�SdSrr')r%Zlr_rangeZdecay_stepsZ
pct_remainingZdecay��lr_endZlr_initr1r#�powerrrr+�sz<get_polynomial_decay_schedule_with_warmup.<locals>.lr_lambda)�defaults�121ValueErrorr-r)rr#r1rCrDrr+rrBr�)get_polynomial_decay_schedule_with_warmup�s122123rG)�namerr#r1r4rDrcCs�t|�}t|}|tjkr&|||d�S|dur<t|�d���|tjkrT||||d�S|durjt|�d���|tjkr�||||||d�S|tjkr�||||||d�S|||||d�S)	au124    Unified API to get any scheduler from its name.125 126    Args:127        name (`str` or `SchedulerType`):128            The name of the scheduler to use.129        optimizer (`torch.optim.Optimizer`):130            The optimizer that will be used during training.131        num_warmup_steps (`int`, *optional*):132            The number of warmup steps to do. This is not required by all schedulers (hence the argument being133            optional), the function will raise an error if it's unset and the scheduler type requires it.134        num_training_steps (`int``, *optional*):135            The number of training steps to do. This is not required by all schedulers (hence the argument being136            optional), the function will raise an error if it's unset and the scheduler type requires it.137        num_cycles (`int`, *optional*):138            The number of hard restarts used in `COSINE_WITH_RESTARTS` scheduler.139        power (`float`, *optional*, defaults to 1.0):140            Power factor. See `POLYNOMIAL` scheduler141        last_epoch (`int`, *optional*, defaults to -1):142            The index of the last epoch when resuming training.143    r!Nz; requires `num_warmup_steps`, please provide that argument.)r#rz= requires `num_training_steps`, please provide that argument.)r#r1r4r)r#r1rDr)r#r1r)r144�TYPE_TO_SCHEDULER_FUNCTIONrrFrrr)rHrr#r1r4rDrZ
schedule_funcrrr�
get_scheduler�s:145146147�148��rJ)r)r)r)r3r)rr)r?r&r)NNrr&r)"�__doc__r6�enumr�typingrrZtorch.optimrZtorch.optim.lr_schedulerr�utilsr	�149get_loggerr�loggerr150r-r"r.r2r(r=r>rGrrrrrrrI�strrJrrrr�<module>sT151	152�153�#�154�$�1553�
�156�