xdecoder/Instruct-X-Decoder
163
1U
2 E$�c� � @ sV d dl Z d dlZd dlZd dlZe�e�Zdd� Zdd� Zdd� Z dd � Z3d4d � Z dS ) � Nc C s� t |t�std��|�� D ]�\}}|�d�}| }|dd� D ].}||krPi ||< || }t |t�s<td��q<|�|d �}|||d < |rt�d|� d|� d||d � �� qdS ) zs5 Load the key, value pairs from config_dict to opt, overriding existing values in opt6 if there is any.7 z"Config must be a Python dictionary�.N�����z0Overriding key needs to be inside a Python dict.z8Overrided z from z to ) �9isinstance�dict� TypeError�items�split�AssertionError�get�logger�warning)�opt� config_dict�k�vZk_parts�pointerZk_partZ ori_value� r �(/data/arXiv/demo/Demo/utils/arguments.py�load_config_dict_to_opt s 1011 r c C s4 i }t | dd��}t�|�}W 5 Q R X t||� |S )z�12 Load opt from the config files, settings in later files can override those in previous files.13 14 Args:15 conf_files: config file path16 17 Returns:18 dict: a dictionary of opt settings19 �utf-8��encoding)�open�yaml� safe_loadr )� conf_filer
�fr r r r �load_opt_from_config_files s20 2122r c s� t jdd�}|jddd� |jdddd � |jd23d d d
� |jddt jd� | sX|�� n|�| �� t� j�}� jr�d�� j�}t �24d|� �� t � |�}t
||� � j�r�t� j�d dks�td��� fdd�tt� j�d �D �}� fdd�tt� j�d �D �}dd� |D �}g }|D ]F}|�d�}|�� } t|�dk�rP| |�d� } �q0|�t| �� �qdd � t|||�D �}t
||� � j�� D ] \}}25|26d k �r�|27||< �q�|� fS ) Nz7MainzTrain: Pretrain or fine-tune models for NLP tasks.)� description�commandz*Command: train/evaluate/train-and-evaluate)�helpz --conf_filesTz)Path(s) to the MainzTrain config file(s).)�requiredr z--config_overrides�*a Override parameters on config with a json style string, e.g. {"<PARAM_NAME_1>": <PARAM_VALUE_1>, "<PARAM_GROUP_2>.<PARAM_SUBGROUP_2>.<PARAM_2>": <PARAM_VALUE_2>}. A key with "." updates the object in the corresponding nested dict. Remember to escape " in command line.)�nargsr z --overridesz9arguments that used to overide the config file in cmdline)r r# � z Command line config overrides: � r z5overides arguments is not paired, required: key valuec s g | ]}� j |d �qS )r% �� overrides��.0�idx�Z cmdline_argsr r �28<listcomp>D s z$load_opt_command.<locals>.<listcomp>c s g | ]}� j |d d �qS )r% � r& r( r+ r r r, E s c S s8 g | ]0}t |�d d��dkr0|�dd��dd�n|�qS )r$ � � �false�False)�len�replace)r) �valr r r r, F s r c S s i | ]\}}}|||��qS r r )r) �x�y�zr r r �29<dictcomp>P s z$load_opt_command.<locals>.<dictcomp>)�argparse�ArgumentParser� add_argument� REMAINDER�30parse_argsr �31conf_filesZconfig_overrides�joinr r �json�loadsr r' r2 r �ranger �copy�pop�append�type�zip�__dict__r ) �args�parserr
Zconfig_overrides_stringr �keys�vals�types�key�eler4 r r+ r �load_opt_command1 s< 32 3334 353637 rP c C s. t |ddd��}tj| |dd� W 5 Q R X d S )N�wr r � )�indent)r r@ �dump�r
r r r r r �save_opt_to_json[ s rV c C s* t |ddd��}t�| |� W 5 Q R X d S )NrQ r r )r r rT rU r r r �save_opt_to_yaml` s rW ) r r@ r9 �logging� getLogger�__name__r r r rP rV rW r r r r �<module> s 38*