Apppublic
procgne/Plonk

sourceHugging Facemitupdated 10mo agoView on Hugging Face
0likes
rootutils__pycache__optimizers.cpython-310.pyc
optimizers.cpython-310.pyc47 linesDownload Raw Back to __pycache__
1o

2c�f&�@s4dZddlZddlmZddlZGdd�de�ZdS)zLamb optimizer.�N)�	Optimizercs.eZdZdZ	d�fdd�	Zd
d3d�Z�ZS)�Lamba�Implements Lamb algorithm.4    It has been proposed in `Large Batch Optimization for Deep Learning: Training BERT in 76 minutes`_.5    Arguments:6        params (iterable): iterable of parameters to optimize or dicts defining7            parameter groups8        lr (float, optional): learning rate (default: 1e-3)9        betas (Tuple[float, float], optional): coefficients used for computing10            running averages of gradient and its square (default: (0.9, 0.999))11        eps (float, optional): term added to the denominator to improve12            numerical stability (default: 1e-8)13        weight_decay (float, optional): weight decay (L2 penalty) (default: 0)14        adam (bool, optional): always use trust ratio = 1, which turns this into15            Adam. Useful for comparison purposes.16    .. _Large Batch Optimization for Deep Learning: Training BERT in 76 minutes:17        https://arxiv.org/abs/1904.0096218    �����MbP?�g�������?g+�����?�:�0�yE>rFcs�d|kstd�|���d|kstd�|���d|dkr"dks,ntd�|d���d|dkr8dksBntd�|d���t||||d	�}||_tt|��||�dS)19NgzInvalid learning rate: {}zInvalid epsilon value: {}rg�?z%Invalid beta parameter at index 0: {}�z%Invalid beta parameter at index 1: {})�lr�betas�eps�weight_decay)�20ValueError�format�dict�adam�superr�__init__)�self�paramsrr	r21rr�defaults��	__class__��5/home/dufour/Documents/diff_plonk/utils/optimizers.pyrsz
Lamb.__init__Nc22Cs�d}|dur	|�}|jD]�}|dD]�}|jdurq|jj}|jr%td��|j|}t|�dkrDd|d<t�|j�|d<t�|j�|d<|d|d}}|d\}	}23|dd	7<|�	|	�j24|d	|	d25�|�	|26�j||d	|27d�d	|	|d}d	|28|d}||}
||}|d}d
|vr�|d
n|ddk}|
|���
|d�}|ddkr�|j29|j|dd30�|r�|jjdd�}|jdd�}t�|�d�t�|�d�||d	�d	�}|js�|s�d	}|jj31|||d32�qq|S)z�Performs a single optimization step.33        Arguments:34            closure (callable, optional): A closure that reevaluates the model35                and returns the loss.36        NrzCLamb does not support sparse gradients, consider SparseAdam instad.r�step�exp_avg�37exp_avg_sqr	r)�alpha)�valuer�layer_adaptationrr38�)�p)�param_groups�grad�data�	is_sparse�RuntimeError�state�len�torch�39zeros_like�mul_�add_�addcmul_�sqrt�add�norm�where�ner)r�closure�loss�groupr r"r&rr�beta1�beta2�bias_correction1�bias_correction2Zexp_avg_hatZexp_avg_sq_hat�	step_sizeZdo_layer_adaptationZ	adam_step�weight_normZ	adam_normZtrust_ratiorrrr)s^4041�42�43��44�;z	Lamb.step)rrrrF)N)�__name__�45__module__�__qualname__�__doc__rr�
__classcell__rrrrrs46�r)r>r(Ztorch.optimr�mathrrrrr�<module>s47