Aluode/PerceptionLabPortable
0
1o
2 .�Yi[2 � @ sf d dl Z ddlmZmZmZ e� rd dlZe�e�ZG dd� d�Z dd� Z3dd � Z G d4d � d e�Z dS ) � N� )� ExplicitEnum�is_torch_available�loggingc @ s� e Zd ZdZdg dfdd�Zd dd�Zdd � Zd5d � Zd d
� Zdd� Z dd� Z6dd� Z dd� Z dd� Z
dd� Zdd� Zd d � Zd d � ZdS )!�DebugUnderflowOverflowa� 7 This debug class helps detect and understand where the model starts getting very large or very small, and more8 importantly `nan` or `inf` weight and activation elements.9 10 There are 2 working modes:11 12 1. Underflow/overflow detection (default)13 2. Specific batch absolute min/max tracing without detection14 15 Mode 1: Underflow/overflow detection16 17 To activate the underflow/overflow detection, initialize the object with the model :18 19 ```python20 debug_overflow = DebugUnderflowOverflow(model)21 ```22 23 then run the training as normal and if `nan` or `inf` gets detected in at least one of the weight, input or output24 elements this module will throw an exception and will print `max_frames_to_save` frames that lead to this event,25 each frame reporting26 27 1. the fully qualified module name plus the class name whose `forward` was run28 2. the absolute min and max value of all elements for each module weights, and the inputs and output29 30 For example, here is the header and the last few frames in detection report for `google/mt5-small` run in fp1631 mixed precision :32 33 ```34 Detected inf/nan during batch_number=035 Last 21 forward frames:36 abs min abs max metadata37 [...]38 encoder.block.2.layer.1.DenseReluDense.wi_0 Linear39 2.17e-07 4.50e+00 weight40 1.79e-06 4.65e+00 input[0]41 2.68e-06 3.70e+01 output42 encoder.block.2.layer.1.DenseReluDense.wi_1 Linear43 8.08e-07 2.66e+01 weight44 1.79e-06 4.65e+00 input[0]45 1.27e-04 2.37e+02 output46 encoder.block.2.layer.1.DenseReluDense.wo Linear47 1.01e-06 6.44e+00 weight48 0.00e+00 9.74e+03 input[0]49 3.18e-04 6.27e+04 output50 encoder.block.2.layer.1.DenseReluDense T5DenseGatedGeluDense51 1.79e-06 4.65e+00 input[0]52 3.18e-04 6.27e+04 output53 encoder.block.2.layer.1.dropout Dropout54 3.18e-04 6.27e+04 input[0]55 0.00e+00 inf output56 ```57 58 You can see here, that `T5DenseGatedGeluDense.forward` resulted in output activations, whose absolute max value was59 around 62.7K, which is very close to fp16's top limit of 64K. In the next frame we have `Dropout` which60 renormalizes the weights, after it zeroed some of the elements, which pushes the absolute max value to more than61 64K, and we get an overflow.62 63 As you can see it's the previous frames that we need to look into when the numbers start going into very large for64 fp16 numbers.65 66 The tracking is done in a forward hook, which gets invoked immediately after `forward` has completed.67 68 By default the last 21 frames are printed. You can change the default to adjust for your needs. For example :69 70 ```python71 debug_overflow = DebugUnderflowOverflow(model, max_frames_to_save=100)72 ```73 74 To validate that you have set up this debugging feature correctly, and you intend to use it in a training that75 may take hours to complete, first run it with normal tracing enabled for one of a few batches as explained in76 the next section.77 78 79 Mode 2. Specific batch absolute min/max tracing without detection80 81 The second work mode is per-batch tracing with the underflow/overflow detection feature turned off.82 83 Let's say you want to watch the absolute min and max values for all the ingredients of each `forward` call of a84 given batch, and only do that for batches 1 and 3. Then you instantiate this class as :85 86 ```python87 debug_overflow = DebugUnderflowOverflow(model, trace_batch_nums=[1, 3])88 ```89 90 And now full batches 1 and 3 will be traced using the same format as explained above. Batches are 0-indexed.91 92 This is helpful if you know that the program starts misbehaving after a certain batch number, so you can93 fast-forward right to that area.94 95 96 Early stopping:97 98 You can also specify the batch number after which to stop the training, with :99 100 ```python101 debug_overflow = DebugUnderflowOverflow(model, trace_batch_nums=[1, 3], abort_after_batch_num=3)102 ```103 104 This feature is mainly useful in the tracing mode, but you can use it for any mode.105 106 107 **Performance**:108 109 As this module measures absolute `min`/``max` of each weight of the model on every forward it'll slow the training110 down. Therefore remember to turn it off once the debugging needs have been met.111 112 Args:113 model (`nn.Module`):114 The model to debug.115 max_frames_to_save (`int`, *optional*, defaults to 21):116 How many frames back to record117 trace_batch_nums(`list[int]`, *optional*, defaults to `[]`):118 Which batch numbers to trace (turns detection off)119 abort_after_batch_num (`int``, *optional*):120 Whether to abort after a certain batch number has finished121 � Nc C sR || _ || _|| _t�g |�| _g | _d| _d| _d| _ d| _122| � � | � � d S )Nr Fz )
�model�trace_batch_nums�abort_after_batch_num� collections�deque�frames�frame� batch_number� total_calls�detected_overflow�prefix�
analyse_model�register_forward_hook)�selfr Zmax_frames_to_saver r123 � r ��E:\DocsHouse\542 percep lab latest\PerceptionLab\PerceptionLab_Portable\python_embed\Lib\site-packages\transformers/debug_utils.py�__init__� s z DebugUnderflowOverflow.__init__c C s0 |d ur | � |� | j�d�| j�� g | _d S �N�124)� expand_framer
�append�joinr )r r r r r �125save_frame� s 126127z!DebugUnderflowOverflow.save_framec C s | j �|� d S �N)r r )r �liner r r r � � z#DebugUnderflowOverflow.expand_framec C s t d�| j�� g | _d S r )�printr r
�r r r r � trace_frames� s 128z#DebugUnderflowOverflow.trace_framesc C s129 g | _ d S r )r
r# r r r �reset_saved_frames� s 130z)DebugUnderflowOverflow.reset_saved_framesc C s` t d| j� �� t dt| j�� d�� t dd�ddd�d�� t d �| j�� t d131� g | _d S ) Nz&132Detected inf/nan during batch_number=zLast z forward frames:�abs min�8� �abs max� metadatar �133 134)r" r �lenr
r r# r r r �dump_saved_frames� s 135z(DebugUnderflowOverflow.dump_saved_framesc C s dd� | j �� D �| _d S )Nc S s i | ]\}}||�qS r r )�.0�name�mr r r �136<dictcomp>� s z8DebugUnderflowOverflow.analyse_model.<locals>.<dictcomp>)r Z
named_modules� module_namesr# r r r r � s z$DebugUnderflowOverflow.analyse_modelc C sn t �|�r| �t||�� t||�rd| _d S d S |d u r*| �dd�d|� �� d S | �dd�d|� �� d S )NT�Nonez>17r( z not a tensor)�torchZ is_tensorr �get_abs_min_max�detect_overflowr )r �var�ctxr r r �analyse_variable� s 137138139�z'DebugUnderflowOverflow.analyse_variablec C s: | � d| j� d| j� d�� | � dd�ddd�d�� d S ) Nr+ z *** Starting batch number=z ***r& r' r( r) r* �r r r r# r r r �batch_start_frame� s z(DebugUnderflowOverflow.batch_start_framec C s" | � | j� d| jd � d�� d S )Nz *** Finished batch number=r z ***140 141r: r# r r r �batch_end_frame� s "z&DebugUnderflowOverflow.batch_end_framec 142 143 C s | � | j� d| j| � d|jj� �� |jdd�D ]144\}}| �||� qt|t�r=t |�D ]\}}| �|d|� d�� q-n| �|d� t|t�rxt |�D ]*\}}t|t�rlt |�D ]\}} | �| d|� d|� d�� qYqL| �|d|� d�� qLn| �|d � | �145� d S )146Nr( F)Zrecursezinput[�]�inputzoutput[z][�output) r r r2 � __class__�__name__Znamed_parametersr9 �147isinstance�tuple� enumerater )148r �moduler> r? r/ �p�i�x�j�yr r r � create_frame� s$ &149� 150151 �� z#DebugUnderflowOverflow.create_framec C s | j �| j� d S r )r �apply�_register_forward_hookr# r r r r � s z,DebugUnderflowOverflow.register_forward_hookc C s |� | j� d S r )r � forward_hook)r rE r r r rM � r! z-DebugUnderflowOverflow._register_forward_hookc C s� d}| j | jv }|r| �� | jdkr| �� | jd7 _|| jkr,| j d7 _ d}| �|||� |r9| �� |r?| �� | jrL|sL| � � t152d��| j d urd| j | j krft153d| j � d| j � d���d S d S ) NFr r Tz�DebugUnderflowOverflow: inf/nan detected, aborting as there is no point running further. Please scroll up above this traceback to see the activation values prior to this event.z'DebugUnderflowOverflow: aborting after z' batches due to `abort_after_batch_num=z` arg) r r r% r r; r rK r$ r r- �154ValueErrorr155 )r rE r> r? Zlast_frame_of_batchZ156trace_moder r r rN � s6 157158159�160���z#DebugUnderflowOverflow.forward_hookr )rA �161__module__� __qualname__�__doc__r r r r$ r% r- r r9 r; r< rK r rM rN r r r r r s u162163 r c C s( | � � }|�� d�d|�� d�d|� �S )Nz8.2er( )�abs�min�max)r7 r8 Zabs_varr r r r5 % s r5 c C sV d}t �| ��� �� rd}t|� d�� t �| ��� �� r&d}t|� d�� |S )a& 164 Report whether the tensor contains any `nan` or `inf` entries.165 166 This is useful for detecting overflows/underflows and best to call right after the function that did some math that167 modified the tensor in question.168 169 This function contains a few other helper features that you can enable and tweak directly if you want to track170 various other things.171 172 Args:173 var: the tensor variable to check174 ctx: the message to print as a context175 176 Return:177 `True` if `inf` or `nan` was detected, `False` otherwise178 FTz has nansz has infs)
r4 �isnan�any�itemr" �isinf�gerS ZnumelrT rU r7 �mean)r7 r8 ZdetectedZn100Zn1000Zn10000r r r r6 * s r6 c @ s e Zd ZdZdZdS )� DebugOptionZunderflow_overflowZtpu_metrics_debugN)rA rP rQ ZUNDERFLOW_OVERFLOWZTPU_METRICS_DEBUGr r r r r\ X s r\ )
r �utilsr r r r4 Z179get_loggerrA �loggerr r5 r6 r\ r r r r �<module> s 180 .