xdecoder/Instruct-X-Decoder
163
1U
2 E$�c� � @ sz d dl Z d dlZd dlZd dlZd dlmZ dd� Zdd� Zdd� Z dd � Z3d4d � Z d d
� Z ddd�Z
dd� Zdd� ZdS )� Nc C s� | � dd�otj�� | d< dtjkr^d| d< d| d< d| d< d | d5< d | d < d | d
< d| d< nHttjd �| d< ttjd �| d< ttjd �| d6< ttjd �| d < | d s�| d dks�td��t�d�| d< n$tj� | d � t�d| d �| d< | S )N�CUDAT�OMPI_COMM_WORLD_SIZEzno MPIZenv_info� �7world_size�8local_sizer �rankZ9local_rankz 127.0.0.1Zmaster_addressZ8673� master_portZOMPI_COMM_WORLD_LOCAL_SIZE�OMPI_COMM_WORLD_RANKZOMPI_COMM_WORLD_LOCAL_RANKz[multi-GPU training without CUDA is not supported since we use NCCL as communication backend�cpu�device�cuda)10�get�torchr � is_available�os�environ�int�AssertionErrorr �11set_device)�opt� r �*/data/arXiv/demo/Demo/utils/distributed.py�init_distributed s& 1213r c C s$ d} dt jkr tt jd �} | dkS )Nr r r )r r r �r r r r �is_main_process% s 14r c C s t �� s dS t �� sdS t �� S )Nr )�distr �is_initialized�get_world_sizer r r r r , s15 r c C s t �� s dS t �� sdS t �� S )Nr )r r r �get_rankr r r r r 3 s16 r c sT t �� s dS t �� sdS t �� } t �� � | dkr4dS � fdd�}|d� |d� dS )zj17 Helper function to synchronize (barrier) among all processes when18 using distributed training19 Nr c sN � | krt jddd�}nt jddd�}t�|| � |�� dkrJt�d� q2d S )Nr r �r r )r �tensorr � broadcast�item�time�sleep)�rr r r r �_send_and_waitI s z#synchronize.<locals>._send_and_waitr )r r r r r )r r& r r r � synchronize; s r' c
C s. t � }|dkr| gS t�| �}tj�|�}t�|��d�}t�|� � g��d�}dd� t20|�D �}t � ||� dd� |D �}t
|�}g }|D ] } |�tj|fd��d�� q�||kr�tj|| fd��d�}21tj||22fdd�}t � ||� g } t||�D ]0\} }|�� �� �� d | � }| �t�|�� q�| S )23z�24 Run all_gather on arbitrary picklable data (not necessarily tensors)25 Args:26 data: any picklable object27 Returns:28 list[data]: list of data gathered from each rank29 r r c S s g | ]}t �d g��d��qS )r r )r � IntTensor�to)�.0�_r r r �30<listcomp>j s z all_gather.<locals>.<listcomp>c S s g | ]}t |�� ��qS r )r r" )r* �sizer r r r, l s )r- r ��dimN)r �pickle�dumpsr � ByteStorage� from_buffer�31ByteTensorr) r( �numel�ranger �32all_gather�max�append�cat�zipr33 �numpy�tobytes�loads)
�datar �buffer�storager r � size_list�max_size� tensor_listr+ �padding� data_listr- r r r r7 W s. 34 r7 Tc C s� t � }|dk r| S t�� �� g }g }t| �� �D ] }|�|� |�| | � q0tj|dd�}tj|dd� t� � dkr�|r�|| }dd� t35||�D �}W 5 Q R X |S )a; 36 Args:37 input_dict (dict): all the values will be reduced38 average (bool): whether to do average or sum39 Reduce the values in the dictionary from all processes so that process with rank40 0 has the averaged results. Returns a dict with the same fields as41 input_dict, after reduction.42 � r r. )�dstc S s i | ]\}}||�qS r r )r* �k�vr r r �43<dictcomp>� s z reduce_dict.<locals>.<dictcomp>) r r �no_grad�sorted�keysr9 �stackr �reducer r; )�44input_dict�averager �names�valuesrI � reduced_dictr r r � reduce_dict� s 4546 rV c C s� t j�� s| S t�� }|dkr4t j| dg dd�}nt j| dg dd�}t j�|d� |�� �� d dkrvt �47d� qV|�� �� � � d d� S )Nr r r r �����) r � distributedr r r r r! r48 r< r# r$ �tolist)r? r Z data_tensorr r r �broadcast_data� s 49 rZ c C s, t � dkr| S | �� } tj| tjjd� | S )Nr )�op)r �cloner �50all_reduce�ReduceOp�SUM)r r r r �51reduce_sum� s52 53r` )T)r r# r r0 Ztorch.distributedrX r r r r r r' r7 rV rZ r` r r r r �<module> s +54