Aluode/PerceptionLabPortable
0
1o
2 .�YifM �
@ sF d dl Z d dlZd dlZd dlZd dlZd dlmZmZmZ d dl m3Z4 d dl m Z d dl m
Z
d dlmZ d dlmZ d dlmZmZmZmZmZmZmZ d dlZed e�Zed5e�Z d d � Z d
e dee gef fdd�Z dde j!e j!dd�deee e e f dee dedeg ef dee" de j#f dd�Z$G dd� de�Z%dS )� N)� ArgumentDefaultsHelpFormatter�ArgumentParser�ArgumentTypeError)�Iterable)�copy)�Enum)�isclass��Path)�Any�Callable�Literal�NewType�Optional�Union�get_type_hints� DataClass�
DataClassTypec C s> t | t�r| S | �� dv rdS | �� dv rdS td| � d���)N)�yes�true�t�y�1T)�no�false�f�n�0FzTruthy value expected: got zJ but expected one of yes/no, true/false, t/f, y/n, 1/0 (case insensitive).)�6isinstance�bool�lowerr )�v� r" ��E:\DocsHouse\542 percep lab latest\PerceptionLab\PerceptionLab_Portable\python_embed\Lib\site-packages\transformers/hf_argparser.py�string_to_bool$ s 7 8�r$ �choices�returnc s dd� | D �� � fdd�S )aN 9 Creates a mapping function from each choices string representation to the actual value. Used to support multiple10 value types for a single argument.11 12 Args:13 choices (list): List of choices.14 15 Returns:16 Callable[[str], Any]: Mapping function from string representation to actual value for each choice.17 c S s i | ]}t |�|�qS r" )�str)�.0�choicer" r" r# �18<dictcomp>< s z-make_choice_type_function.<locals>.<dictcomp>c s � � | | �S �N)�get)�arg�Z
str_to_choicer" r# �<lambda>= s z+make_choice_type_function.<locals>.<lambda>r" )r% r" r. r# �make_choice_type_function1 s r0 )�aliases�help�default�default_factory�metadatar1 r2 r3 r4 r5 c K sD |du ri }| dur| |d< |dur||d< t jd|||d�|��S )a� Argument helper enabling a concise syntax to create dataclass fields for parsing with `HfArgumentParser`.19 20 Example comparing the use of `HfArg` and `dataclasses.field`:21 ```22 @dataclass23 class Args:24 regular_arg: str = dataclasses.field(default="Huggingface", metadata={"aliases": ["--example", "-e"], "help": "This syntax could be better!"})25 hf_arg: str = HfArg(default="Huggingface", aliases=["--example", "-e"], help="What a nice syntax!")26 ```27 28 Args:29 aliases (Union[str, list[str]], optional):30 Single string or list of strings of aliases to pass on to argparse, e.g. `aliases=["--example", "-e"]`.31 Defaults to None.32 help (str, optional): Help string to pass on to argparse that can be displayed with --help. Defaults to None.33 default (Any, optional):34 Default value for the argument. If not default or default_factory is specified, the argument is required.35 Defaults to dataclasses.MISSING.36 default_factory (Callable[[], Any], optional):37 The default_factory is a 0-argument function called to initialize a field's value. It is useful to provide38 default values for mutable types, e.g. lists: `default_factory=list`. Mutually exclusive with `default=`.39 Defaults to dataclasses.MISSING.40 metadata (dict, optional): Further metadata to pass on to `dataclasses.field`. Defaults to None.41 42 Returns:43 Field: A `dataclasses.Field` with the desired properties.44 Nr1 r2 )r5 r3 r4 r" )� dataclasses�field)r1 r2 r3 r4 r5 �kwargsr" r" r# �HfArg@ s $r9 c s e Zd ZU dZee ed< d deeeee f f� fdd�
Z e45de de j
fdd ��Zd46efd d �Z
d deedf fdd�Zd deeef dedeedf fdd�Z
d deeejf dedeedf fdd�Z
d deeejf dedeedf fdd �Z� Z S ) �HfArgumentParsera 47 This subclass of `argparse.ArgumentParser` uses type hints on dataclasses to generate arguments.48 49 The class is designed to play well with the native argparse. In particular, you can add more (non-dataclass backed)50 arguments to the parser after initialization and you'll get the output back after parsing as an additional51 namespace. Optional: To create sub argument groups use the `_argument_group_name` attribute in the dataclass.52 53 Args:54 dataclass_types (`DataClassType` or `Iterable[DataClassType]`, *optional*):55 Dataclass type, or list of dataclass types for which we will "fill" instances with the parsed args.56 kwargs (`dict[str, Any]`, *optional*):57 Passed to `argparse.ArgumentParser()` in the regular way.58 �dataclass_typesNc st |d u rg }nt |t�s|g}d|vrt|d< t� jdi |�� t�|�r(|g}t|�| _| jD ]}| � |� q0d S )NZformatter_classr" )59r r r �super�__init__r6 � is_dataclass�listr; �_add_dataclass_arguments)�selfr; r8 �dtype�� __class__r" r# r= � s 60616263 �zHfArgumentParser.__init__�parserr7 c C s� d|j � �g}d|j v r|�d|j �dd�� �� |j�� }t|jt�r(td��|� dg �}t|t�r6|g}t64|jd|j�}|t u sMt t
d�r�t|t
j�r�t|jjvrlt|jj�dksctd �|jjvrltd |j � d65���td �|jjvr�|jjd tu r�|jjd n|jjd |_t66|jd|j�}n$t|jjvr�td |jjd �r�|jjd n|jjd |_t67|jd|j�}i }|tu s�t|jt�r�t|jt�r�|tu r�|jj|d
< n68dd� |jD �|d
< t|d
�|d< |jtjur�|j|d< n�d|d< n�|jtu �s|jtt k�r8t|�}t|d< |jtu �s |jd u�r7|jtju�r7|jtju �r(dn|j}||d< d|d< d|d< nUt |��rit|t ��ri|jjd |d< d|d< |j tju�r]|� � |d< n0|jtju �rhd|d< n$|j|d< |jtju�r{|j|d< n|j tju�r�|� � |d< nd|d< | j g |�|�R i |�� |jdu �r�|jtu �s�|jtt k�r�d|d< | j d|j � �d|j �dd�� �fd|j d �|�� d S d S d S ) Nz--�_�-zpUnresolved type detected, which should have been done with the help of `typing.get_type_hints` method by defaultr1 �69__origin__� UnionType� z�Only `Union[X, NoneType]` (i.e., `Optional[X]`) is allowed for `Union` because the argument parser only supports one type per argument. Problem encountered in field 'z'.� r r% c S s g | ]}|j �qS r" )�value)r( �xr" r" r# �70<listcomp>� s z;HfArgumentParser._parse_dataclass_field.<locals>.<listcomp>�typer3 T�requiredF�?�nargs�const�+z--no_z--no-� store_false)�action�dest) �name�append�replacer5 r r rO r' � RuntimeError�pop�getattrr �hasattr�typesrI �__args__�len�71ValueErrorr r
�72issubclassr r0 r3 r6 �MISSINGr r$ r r? r4 � add_argument)rE r7 Z long_optionsr8 r1 Z origin_typeZ bool_kwargsr3 r" r" r# �_parse_dataclass_field� s� 7374 � 75 ��* (� 76 &��77 (78��79��z'HfArgumentParser._parse_dataclass_fieldrB c C s| t |d�r | �|j�}n| }zt|�}W n ty# td|� d���w t�|�D ]}|js/q)||j |_80| � ||� q)d S )N�_argument_group_namezType resolution failed for z�. Try declaring the class in global scope or removing line of `from __future__ import annotations` which opts in Postponed Evaluation of Annotations (PEP 563)) r^ �add_argument_grouprg r � NameErrorr[ r6 �fields�initrX rO rf )rA rB rE Z81type_hintsr7 r" r" r# r@ � s 82 83�� �z)HfArgumentParser._add_dataclass_argumentsFTr& .c s� |s |s |r{t tj�r{g }|r|�t|�� n|r+t tj�r+|�ttjd ��d�� |rWt� }|j|tdd� |j |d�\}}t84|�� |� d�d�} | rW|�
dd � | D �� g }85|D ]} | �� ri|86| �� �� 7 }87q[|durr|88| n|89tjd90d� }| j |d�\} }
g }| jD ]/}d d � t�|�D �� � fd
d�t91| ��� D �}� D ]}t| |� q�|di |��}|�|� q�t | j�dkr�|�| � |r�g |�|
�R S |
r�td|
� ���g |�R S )a% 92 Parse command-line args into instances of the specified dataclass types.93 94 This relies on argparse's `ArgumentParser.parse_known_args`. See the doc at:95 docs.python.org/3/library/argparse.html#argparse.ArgumentParser.parse_args96 97 Args:98 args:99 List of strings to parse. The default is taken from sys.argv. (same as argparse.ArgumentParser)100 return_remaining_strings:101 If true, also return a list of remaining argument strings.102 look_for_args_file:103 If true, will look for a ".args" file with the same base name as the entry point script for this104 process, and will append its potential content to the command line args.105 args_filename:106 If not None, will uses this file instead of the ".args" file specified in the previous argument.107 args_file_flag:108 If not None, will look for a file in the command-line args specified with this flag. The flag can be109 specified multiple times and precedence is determined by the order (last one wins).110 111 Returns:112 Tuple consisting of:113 114 - the dataclass instances in the same order as they were passed to the initializer.abspath115 - if applicable, an additional namespace for more (non-dataclass backed) arguments added to the parser116 after initialization.117 - The potential list of remaining argument strings. (same as argparse.ArgumentParser.parse_known_args)118 r z.argsrY )rO rV )�argsrG Nc S s g | ]}t |��qS r" r )r( �pr" r" r# rN H s z@HfArgumentParser.parse_args_into_dataclasses.<locals>.<listcomp>rK c S � h | ]}|j r|j�qS r" �rk rX �r( r r" r" r# � <setcomp>U � z?HfArgumentParser.parse_args_into_dataclasses.<locals>.<setcomp>c � i | ] \}}|� v r||�qS r" r" �r( �kr! ��keysr" r# r* V � z@HfArgumentParser.parse_args_into_dataclasses.<locals>.<dictcomp>z?Some specified arguments are not used by the HfArgumentParser: r" )ra �sys�argvrY r119 � with_suffixr re r' Zparse_known_args�varsr, �lstrip�extend�exists� read_text�splitr; r6 rj �items�delattr�__dict__rb )rA rl Zreturn_remaining_stringsZlook_for_args_fileZ
args_filenameZargs_file_flagZ120args_filesZargs_file_parser�cfgZcmd_args_file_pathsZ file_argsZ args_file� namespaceZremaining_args�outputsrB �inputsru �objr" rv r# �parse_args_into_dataclasses sF %�"121 122123z,HfArgumentParser.parse_args_into_dataclassesrl �allow_extra_keysc s� t |�� �}g }| jD ]*}dd� t�|�D �� � fdd�|�� D �}|�|�� � |di |��}|�|� q |sC|rCtdt |�� ���t124|�S )a< 125 Alternative helper method that does not use `argparse` at all, instead uses a dict and populating the dataclass126 types.127 128 Args:129 args (`dict`):130 dict containing config values131 allow_extra_keys (`bool`, *optional*, defaults to `False`):132 Defaults to False. If False, will raise an exception if the dict contains keys that are not parsed.133 134 Returns:135 Tuple consisting of:136 137 - the dataclass instances in the same order as they were passed to the initializer.138 c S rn r" ro rp r" r" r# rq y rr z.HfArgumentParser.parse_dict.<locals>.<setcomp>c rs r" r" rt rv r" r# r* z rx z/HfArgumentParser.parse_dict.<locals>.<dictcomp>z0Some keys are not used by the HfArgumentParser: Nr" ) �setrw r; r6 rj r� �difference_updaterY rb �sorted�tuple)rA rl r� Z unused_keysr� rB r� r� r" rv r# �139parse_dictf s 140 zHfArgumentParser.parse_dict� json_filec C sT t t|�dd��}t�|�� �}W d � n1 sw Y | j||d�}t|�S )at 141 Alternative helper method that does not use `argparse` at all, instead loading a json file and populating the142 dataclass types.143 144 Args:145 json_file (`str` or `os.PathLike`):146 File name of the json file to parse147 allow_extra_keys (`bool`, *optional*, defaults to `False`):148 Defaults to False. If False, will raise an exception if the json file contains keys that are not149 parsed.150 151 Returns:152 Tuple consisting of:153 154 - the dataclass instances in the same order as they were passed to the initializer.155 zutf-8)�encodingN�r� )�openr156 �json�loads�readr� r� )rA r� r� Zopen_json_file�datar� r" r" r# �parse_json_file� s157 �z HfArgumentParser.parse_json_file� yaml_filec C s$ | j t�t|��� �|d�}t|�S )at 158 Alternative helper method that does not use `argparse` at all, instead loading a yaml file and populating the159 dataclass types.160 161 Args:162 yaml_file (`str` or `os.PathLike`):163 File name of the yaml file to parse164 allow_extra_keys (`bool`, *optional*, defaults to `False`):165 Defaults to False. If False, will raise an exception if the json file contains keys that are not166 parsed.167 168 Returns:169 Tuple consisting of:170 171 - the dataclass instances in the same order as they were passed to the initializer.172 r� )r� �yamlZ safe_loadr173 r� r� )rA r� r� r� r" r" r# �parse_yaml_file� s z HfArgumentParser.parse_yaml_filer+ )NFTNN)F) �__name__�174__module__� __qualname__�__doc__r r �__annotations__r r r= � staticmethodr r6 �Fieldrf r@ r� r r� �dictr' r r r� �os�PathLiker� r� �
__classcell__r" r" rC r# r: o s@ 175 $i�176177�(V � ��178179�� ��180�r: )&r6 r� r� ry r_ �argparser r r �collections.abcr r �enumr �inspectr �pathlibr181 �typingr r r
r r r r r� r r r$ r? r' r0 rd r� r� r9 r: r" r" r" r# �<module> sH $182183
����184��185�/