Apppublic
Aluode/PerceptionLabPortable

sourceHugging Faceupdated 9mo agoView on Hugging Face
0likes
rootpython_embedLibsite-packagestransformers__pycache__tokenization_utils.cpython-310.pyc
tokenization_utils.cpython-310.pyc397 linesDownload Raw Back to __pycache__
1o

2.�Yi���@s*dZddlZddlZddlZddlZddlmZddlmZm	Z	m3Z4mZddlm
Z
mZmZmZmZmZmZmZmZmZmZmZmZddlmZmZmZmZe�e �Z!dZ"d	Z#d5Z$Gdd�d�Z%Gd
d�de%�Z&dd�Z'dd�Z(dd�Z)dd�Z*dd�Z+de,e-de-fdd�Z.ee�Gdd�de��Z/dS)z�6Tokenization classes for python tokenizers. For fast tokenizers (provided by HuggingFace's tokenizers library) see7tokenization_utils_fast.py8�N)�OrderedDict)�Any�Optional�Union�overload�)
�ENCODE_KWARGS_DOCSTRING�'ENCODE_PLUS_ADDITIONAL_KWARGS_DOCSTRING�INIT_TOKENIZER_DOCSTRING�9AddedToken�
BatchEncoding�EncodedInput�EncodedInputPair�PreTokenizedInput�PreTokenizedInputPair�PreTrainedTokenizerBase�	TextInput�
TextInputPair�TruncationStrategy)�PaddingStrategy�10TensorType�add_end_docstrings�loggingzspecial_tokens_map.jsonzadded_tokens.jsonztokenizer_config.jsonc@sLeZdZdZdd�Zdd�Zdefdd�Zd	ed11eefdd�Z	d
d�Z12dS)�Triez�13    Trie in Python. Creates a Trie out of a list of words. The trie is used to split on `added_tokens` in one pass14    Loose reference https://en.wikipedia.org/wiki/Trie15    cGs"i|_t�|_d|_|j|�dS)N�)�data�set�_tokens�_termination_char�update��self�args�r#��E:\DocsHouse\542 percep lab latest\PerceptionLab\PerceptionLab_Portable\python_embed\Lib\site-packages\transformers/tokenization_utils.py�__init__:sz
Trie.__init__cGst|�D]}|�|�qdS)z�16        Updates the Trie with new tokens provided as arguments.17 18        Args:19            *args: Variable number of words to be added to the Trie.20        N)�tuple�add)r!r"�tokenr#r#r$r@s�zTrie.update�wordcCsJ|sdS|j�|�|j}|D]}|�|i�||<||}qd||j<dS)u�21        Passes over every char (utf-8 char) on word and recursively adds it to the internal `data` trie representation.22        The special key `""` in `self._termination_char` is used to represent termination.23 24        This function is idempotent, adding twice the same word will leave the trie unchanged25 26        Example:27 28        ```python29        >>> trie = Trie()30        >>> trie.add("Hello 友達")31        >>> trie.data32        {"H": {"e": {"l": {"l": {"o": {" ": {"友": {"達": {"": 1}}}}}}}}}33 34        >>> trie.add("Hello")35        >>> trie.data36        {"H": {"e": {"l": {"l": {"o": {"": 1, " ": {"友": {"達": {"": 1}}}}}}}}}37        ```38        Nr)rr'r�39setdefaultr)r!r)�ref�charr#r#r$r'Js40zTrie.add�text�returncCs�t�}dg}d}t|�D]�\}}|r||krqt�}d}|��D]�\}	}41d|42vr�|��D]V\}}||	kr6nM||	krC|d}
|d}n|}
|}|
t|�krQ||
nd}d|vr]|}	|
}|
}||vr�||}|
d7}
d|vrs|}	|
}|
}|
t|�krzn||
}||vsaq,|�|	�|�|�d}n||43vr�|44|}45|46||	<q |�|	�q |r�i}n|D]}	||	=q�||kr�||jvr�|j|||<q|��D]\}	}47d|48vr�t|�}|�|	�|�|�nq�|�||�S)aY49        Will look for the words added to the trie within `text`. Output is the original string split along the50        boundaries of the words found.51 52        This trie will match the longest possible word first !53 54        Example:55 56        ```python57        >>> trie = Trie()58        >>> trie.split("[CLS] This is a extra_id_100")59        ["[CLS] This is a extra_id_100"]60 61        >>> trie.add("[CLS]")62        >>> trie.add("extra_id_1")63        >>> trie.add("extra_id_100")64        >>> trie.split("[CLS] This is a extra_id_100")65        ["[CLS]", " This is a ", "extra_id_100"]66        ```67        rFrrNT)	r�	enumerater�items�len�appendr'r�cut_text)r!r-Zstates�offsets�skip�currentZcurrent_char�	to_remove�reset�startZtrie_pointerZ	lookstartZlooktrie_pointerZlookahead_index�endZ	next_charr#r#r$�splitist!	68��697071�7273�74z75Trie.splitcCsX|�t|��g}d}|D]}||krt�d�q
||krq
|�|||��|}q
|S)NrzbThere was a bug in Trie algorithm in tokenization. Attempting to recover. Please report it anyway.)r2r1�logger�error)r!r-r4�tokensr9r:r#r#r$r3s�z
Trie.cut_textN)�__name__�76__module__�__qualname__�__doc__r%r�strr'�listr;r3r#r#r#r$r4s77rcsNeZdZ�fdd�Zdefdd�Zdedefdd	�Zd78edefdd�Z	�Z79S)
�ExtensionsTriecst�j|�dS�N)�superr%r ��	__class__r#r$r%szExtensionsTrie.__init__�prefixcs&|���}|�|�}�fdd�|D�S)aC80        Generates all extensions of a given prefix token in the Trie.81 82        Example:83 84        ```python85        >>> trie = Trie()86        >>> trie.add("apple")87        >>> trie.add("app")88        >>> trie.add("application")89        >>> trie.extensions("app")90        ['app', 'apple', 'application']91        ```92        c�g|]}�|�qSr#r#��.0r(�rJr#r$�93<listcomp>0�z-ExtensionsTrie.extensions.<locals>.<listcomp>)�	_get_node�_collect_tokens)r!rJZprefix_node�retr#rNr$�94extensionss9596zExtensionsTrie.extensionsr(r.cCs*|j}|D]
}||vr|S||}q|S)a97        Retrieves the node corresponding to the given token in the Trie.98 99        Args:100            token (str): The token for which the corresponding node needs to be retrieved.101 102        Returns:103            dict: The node in the Trie corresponding to the given token.104        )r)r!r(�noder,r#r#r$rQ2s105106�zExtensionsTrie._get_noderUcsX|j|vr	|jgng}|��D]\�}�|jkr)|�|�}|��fdd�|D��q|S)a107        Generates all tokens in the Trie starting from a given node.108 109        Args:110            node (dict): The node in the Trie from which tokens need to be generated.111 112        Returns:113            list: List of tokens generated from the given node.114        crKr#r#)rMZsubtoken�r(r#r$rORrPz2ExtensionsTrie._collect_tokens.<locals>.<listcomp>)rr0rR�extend)r!rUr>Zsubtrie_headZ	subtokensr#rVr$rRDs115116117�zExtensionsTrie._collect_tokens)r?r@rAr%rCrT�dictrQrDrR�
__classcell__r#r#rHr$rEs118rEcCs>|dks|dks|dks|dkrdSt�|�}|dkrdSdS)z0Checks whether `char` is a whitespace character.� �	�119�
T�ZsF)�unicodedata�category�r,�catr#r#r$�_is_whitespaceVs 120rccCs8|dks|dks|dkrdSt�|�}|�d�rdSdS)z-Checks whether `char` is a control character.r[r\r]F�CT)r_r`�121startswithrar#r#r$�_is_controlbs122123rfcCsht|�}|dkr|dks$|dkr|dks$|dkr|dks$|dkr&|dkr&d	St�|�}|�d124�r2d	SdS)z1Checks whether `char` is a punctuation character.�!�/�:�@�[�`�{�~T�PF)�ordr_r`re)r,�cprbr#r#r$�_is_punctuationns@125126rrcC�$|d}tt|�t|�Bt|�B�S)zcChecks whether the last character in text is one of a punctuation, control or whitespace character.�������boolrfrrrc)r-Z	last_charr#r#r$�_is_end_of_word}�rwcCrs)zdChecks whether the first character in text is one of a punctuation, control or whitespace character.rru)r-Z127first_charr#r#r$�_is_start_of_word�rxry�128token_list�	new_tokencCs8t�||�}|t|�kr|||krdS|�||�dS)zm129    Inserts one token to an ordered list if it does not already exist. Note: token_list must be sorted.130    N)�bisect�bisect_leftr1�insert)rzr{Z
insertion_idxr#r#r$�!_insert_one_token_to_ordered_list�src's�eZdZdZ�fdd�Zedefdd��Zedefdd��Z	ede131eeffd	d132��Zede133ee
ffdd��Zejd
e134eee
effde135ee
ffdd��Zde136eeffdd�Zdd�Zdd�Zd`deeeee
fdedefdd�Zdadeeefdd�Zd`dedefdd �Zd!edeefd"d#�Zd$d%�Zd&eeeefdeeeeffd'd(�Zd)d*�Zd+d,�Zdd-ej e!j"dd.ddddddddddd-fd!eee#e$fd/eeee#e$fd0ed1ed2e!d3eed4ed5ed6eed7eed8eeee%fd9eed:eed;ed<ed=ed>ed?ede&f&d@dA�Z'd-ej e!j"dd.ddddddddddd-dfdBeeeee(ee#ee)ee$ee*fd0ed1ed2e!d3eed4ed5ed6eed7eed8eeee%fd9eed:eed;ed<ed=ed>ed?edCede&f&dDdE�Z+e,e-e.�d-ej e!j"dd.ddddddddd-dfdFeee)e/eedffd0ed1ed2e!d3eed4ed6eed7eed8eed9eed:eed;ed<ed>ed?edCede&f"dGdH��Z0	d`d!ed5ede/ee137ee1fffdIdJ�Z2	dbdKedLeedMedeef�fdNdO�
Z3e4d`dPedQedefdRdS��Z5e4d`dPeedQedeefdTdS��Z5	d`dPeeeefdQedeeeeffdUdS�Z5dVedefdWdX�Z6d&eedefdYdZ�Z7			-dcd[eeeefdQed\eed]edef138d^d_�Z8�Z9S)d�PreTrainedTokenizera139    Base class for all slow tokenizers.140 141    Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`].142 143    Handle all the shared methods for tokenization and special tokens as well as methods downloading/caching/loading144    pretrained tokenizers as well as adding tokens to the vocabulary.145 146    This class also contain the added tokens in a unified way on top of all tokenizers so we don't have to handle the147    specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...).148    cs|t��_t�d�si�_�j�|�di��dd��j��D��_t�j	d149i|���j150�fdd��jD�dd�d	�_dS)N�_added_tokens_decoder�added_tokens_decodercS�i|]\}}|j|�qSr#��content�rM�v�kr#r#r$�151<dictcomp>��z0PreTrainedTokenizer.__init__.<locals>.<dictcomp>csg|]	}|�jvr|�qSr#��_added_tokens_encoderrL�r!r#r$rO�sz0PreTrainedTokenizer.__init__.<locals>.<listcomp>T)�special_tokensFr#)
r�tokens_trie�hasattrr�r�popr0r�rGr%�_add_tokensZall_special_tokens_extended�_decode_use_source_tokenizer)r!�kwargsrHr�r$r%�s152�153zPreTrainedTokenizer.__init__r.cCsdS�NFr#r�r#r#r$�is_fast�szPreTrainedTokenizer.is_fastcC�t�)zP154        `int`: Size of the base vocabulary (without the added tokens).155        ��NotImplementedErrorr�r#r#r$�156vocab_size�szPreTrainedTokenizer.vocab_sizecCs dd�t|j��dd�d�D�S)z�157        Returns the sorted mapping from string to index. The added tokens encoder is cached for performance158        optimisation in `self._added_tokens_encoder` for the slow tokenizers.159        cSr�r#r�r�r#r#r$r��r�z<PreTrainedTokenizer.added_tokens_encoder.<locals>.<dictcomp>cS�|dS�Nrr#��itemr#r#r$�<lambda>��z:PreTrainedTokenizer.added_tokens_encoder.<locals>.<lambda>��key)�sortedr�r0r�r#r#r$�added_tokens_encoder�s z(PreTrainedTokenizer.added_tokens_encodercCstt|j��dd�d��S)z�160        Returns the added tokens in the vocabulary as a dictionary of index to AddedToken.161 162        Returns:163            `dict[str, int]`: The added tokens.164        cSr�r�r#r�r#r#r$r��r�z:PreTrainedTokenizer.added_tokens_decoder.<locals>.<lambda>r�)rXr�r�r0r�r#r#r$r��sz(PreTrainedTokenizer.added_tokens_decoder�valuec	Cs�|��D]9\}}t|ttf�rt|t�s(td|j|jf�dttttff����t|t�r1t|�n||j|<||j	t|�<q|�165�dS)Nz;The provided `added_tokens_decoder` has an element of type z, should be a dict of )r0�166isinstancerCr�int�	TypeErrorrIrr�r��_update_total_vocab_size)r!r��indexr(r#r#r$r��s"�cC�|jS)aX167        Returns the added tokens in the vocabulary as a dictionary of token to index. Results might be different from168        the fast call because for now we always add the tokens even if they are already in the vocabulary. This is169        something we should change.170 171        Returns:172            `dict[str, int]`: The added tokens.173        r�r�r#r#r$�get_added_vocab�s	z#PreTrainedTokenizer.get_added_vocabcCr�)zD174        Size of the full vocabulary with the added tokens.175        )�total_vocab_sizer�r#r#r$�__len__�szPreTrainedTokenizer.__len__cCst|���|_dS)a!176        Update the size of the full vocabulary with the added tokens. Counts the `keys` and not the `values` because177        otherwise if there is a hole in the vocab, we will add tokenizers at a wrong index. This operation is slow and178        is only updated when adding tokens.179        N)r1�	get_vocabr�r�r#r#r$r��sz,PreTrainedTokenizer._update_total_vocab_sizeF�180new_tokensr�c	Cszd}|dur|S|����}t|�}|D]�}t|ttf�s*td|�dt|��d���t|�dkr1qt|t�rN||jvr<q||j	vpB|}t|dd||d�}n|rY|�181d	|jd182��||jvr_q|j
sq|jrqt|dd�rq|j��|_|j|vr�||}|||j<|d7}n||j}|j
r�t|�|j	vr�|jd
�|�||j|<||j|j<|jr�t�d|�d��q|��|��|S)a�183        Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to184        it with indices starting from length of the current vocabulary. Special tokens are sometimes already in the185        vocab which is why they have to be handled specifically.186 187        Args:188            new_tokens (`list[str]`or `list[tokenizers.AddedToken]`):189                Token(s) to add in vocabulary. A token is counted as added if it's not already in the vocabulary190                (tested by checking if the tokenizer assign the index of the `unk_token` to them). If a token is part191                of the vocabulary then we simply mark this token as an `AddedToken` which allows to control the192                stripping and normalization of this token. This is NOT possible in `tokenizers`.193            special_tokens (`bool`, *optional*, defaults to `False`):194                Whether or not the tokens should be added as special tokens.195 196        Returns:197            `int`: The number of tokens actually added to the vocabulary.198 199        Examples:200 201        ```python202        # Let's see how to increase the vocabulary of Bert model and tokenizer203        tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-uncased")204        model = BertModel.from_pretrained("google-bert/bert-base-uncased")205 206        num_added_toks = tokenizer.add_tokens(["new_tok1", "my_new-tok2"])207        print("We have added", num_added_toks, "tokens")208        # Note: resize_token_embeddings expects to receive the full size of the new vocabulary, i.e. the length of the tokenizer.209        model.resize_token_embeddings(len(tokenizer))210        ```rNzToken z is not a string but a �.rF)�rstrip�lstrip�211normalized�specialT)r�r��
do_lower_caser�additional_special_tokenszAdding z to the vocabulary)r��copyr1r�rCrr��typer��all_special_tokens�__setstate__r�r�r��getattrr��lowerZ_special_tokens_mapr2�verboser<�info�_update_trier�)	r!r�r�Zadded_tokensZ
current_vocabZnew_idxr(Z212is_specialZtoken_indexr#r#r$r�sL213214�215216217218219220�zPreTrainedTokenizer._add_tokensN�unique_no_split_tokenscCsV|j��D]}|j|jjvr|j�|j�q|pgD]}||jjvr(|j�|�qdSrF)r��valuesr�r�rr')r!r�r(r#r#r$r�Ns���z PreTrainedTokenizer._update_trie�paircCs$g}g}t|�||r|��Sd��S)aG221        Returns the number of added tokens when encoding a sequence with special tokens.222 223        <Tip>224 225        This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put226        this inside your training loop.227 228        </Tip>229 230        Args:231            pair (`bool`, *optional*, defaults to `False`):232                Whether the number of added tokens should be computed in the case of a sequence pair or a single233                sequence.234 235        Returns:236            `int`: Number of special tokens added to sequences.237        N)r1Z build_inputs_with_special_tokens)r!r��token_ids_0�token_ids_1r#r#r$�num_special_tokens_to_addVsz-PreTrainedTokenizer.num_special_tokens_to_addr-cKs$|�d|j�}|j|fi|��\}}|rt�d|�d��t|d�rM|jrMdd�|jD�}|dd�|j�	�D�7}dd	�238|�d239d}t�|dd
�|�}|rUg}|g}n|j
��}|j�|�}t|�D]�\}}	|	|vr�|j�|j
|	d�}240|dkr�||dnd}|t|�dkr�||dnd}t|241t�r�|242jr�|r�|��||d<|243jr�|r�|��||d<|244jr�|r�|ddkr�||d|	7<d||<qd|245jr�|r�|ddkr�|	||d||d<d||<qdt|246�dt|247�����qdg}
|D]}	|	s�q�|	|v�r|
�|	�q�|
�|�|	��q�|
S)a$248        Converts a string into a sequence of tokens, using the tokenizer.249 250        Split in words for word-based vocabulary or sub-words for sub-word-based vocabularies251        (BPE/SentencePieces/WordPieces). Takes care of added tokens.252 253        Args:254            text (`str`):255                The sequence to be encoded.256            **kwargs (additional keyword arguments):257                Passed along to the model-specific `prepare_for_tokenization` preprocessing method.258 259        Returns:260            `list[str]`: The list of tokens.261        �split_special_tokenszKeyword arguments z not recognized.r�cSsg|]}t�|��qSr#)�re�escape�rMZs_tokr#r#r$rO�sz0PreTrainedTokenizer.tokenize.<locals>.<listcomp>cSs$g|]}|js|jrt�|j��qSr#)r�r�r�r�r�r�r#r#r$rO�s��262��(�|z)|z(.+?)cSs|��dp
|��d��S)Nrr)�groupsr�)�mr#r#r$r��sz.PreTrainedTokenizer.tokenize.<locals>.<lambda>NrrrtrZrzy cannot be tokenized because it was not properly added to the tokenizer. This means that it is not an `AddedToken` but a )r�r��prepare_for_tokenizationr<�warningr�r�r�r�r��joinr��subr��keysr�r;r/�getr1r�rr�r�Zsingle_word�263ValueErrorr�r2rW�	_tokenize)r!r-r�r�Zescaped_special_toks�patternZno_split_tokenr>�ir(Ztok_extended�left�rightZtokenized_textr#r#r$�tokenizems^�264 265266267268����269zPreTrainedTokenizer.tokenizecKr�)a270        Converts a string into a sequence of tokens (string), using the tokenizer. Split in words for word-based271        vocabulary or sub-words for sub-word-based vocabularies (BPE/SentencePieces/WordPieces).272 273        Do NOT take care of added tokens.274        r�)r!r-r�r#r#r$r��szPreTrainedTokenizer._tokenizer>cCsB|durdSt|t�r|�|�Sg}|D]275}|�|�|��q|S)aT276        Converts a token string (or a sequence of tokens) in a single integer id (or a sequence of ids), using the277        vocabulary.278 279        Args:280            tokens (`str` or `list[str]`): One or several token(s) to convert to token id(s).281 282        Returns:283            `int` or `list[int]`: The token id or list of token ids.284        N)r�rC�#_convert_token_to_id_with_added_vocr2)r!r>�idsr(r#r#r$�convert_tokens_to_ids�s285286z)PreTrainedTokenizer.convert_tokens_to_idscCs*|durdS||jvr|j|S|�|�SrF)r��_convert_token_to_id�r!r(r#r#r$r��s287288289290z7PreTrainedTokenizer._convert_token_to_id_with_added_voccCr�rFr�r�r#r#r$r���z(PreTrainedTokenizer._convert_token_to_idTr�	text_pair�add_special_tokens�padding_strategy�truncation_strategy�291max_length�stride�is_split_into_words�pad_to_multiple_of�padding_side�return_tensors�return_token_type_ids�return_attention_mask�return_overflowing_tokens�return_special_tokens_mask�return_offsets_mapping�
return_lengthr�cs����fdd�}|rtd��||�}|dur||�nd}�j|fid|�d|�d|j�d|j�d|�d	|�d292|	�d|293�d|�d
d�d|
�d|�d|�d|�d|�d|��S)Ncs�t|t�r�j|fi���}��|�St|ttf�rBt|�dkrBt|dt�rB�r=ttj��fdd�|D���}��|�S��|�St|ttf�rXt|�dkrXt|dt	�rX|S�rbt294d|�d���t295d|�d���)Nrc3�&�|]}�j|fddi���VqdS�r�TN�r��rM�t�r�r!r#r$�	<genexpr>��$zJPreTrainedTokenizer._encode_plus.<locals>.get_input_ids.<locals>.<genexpr>zInput z] is not valid. Should be a string or a list/tuple of strings when `is_split_into_words=True`.zW is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers.�r�rCr�r�rDr&r1�	itertools�chainr�r��r-r>�r�r�r!r#r$�
get_input_ids�s&296297(�298299(300�301�z7PreTrainedTokenizer._encode_plus.<locals>.get_input_idsareturn_offset_mapping is not available when using Python tokenizers. To use this feature, change your tokenizer to one deriving from transformers.PreTrainedTokenizerFast. More information on available tokenizers at https://github.com/huggingface/transformers/pull/2674�pair_idsr��padding�302truncationr�r�r�r�r��prepend_batch_axisTr�r�r�r�r�r�)r��prepare_for_modelr�)r!r-r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r��	first_ids�303second_idsr#r�r$�_encode_plus�sT���������	�304���
�����z PreTrainedTokenizer._encode_plus�batch_text_or_text_pairsr�cs����fdd�}|rtd��g}|D]3}t|ttf�r&�r,t|dttf�s,|d}}n|\}}||�}|dur<||�nd}|�||f�q�j|f|||||||	|||
|||305||d��}t|�S)Ncs�t|t�r�j|fi���}��|�St|ttf�rBt|�dkrBt|dt�rB�r=ttj��fdd�|D���}��|�S��|�St|ttf�rXt|�dkrXt|dt	�rX|St306d��)Nrc3r�r�r�r�r�r#r$r�[r�zPPreTrainedTokenizer._batch_encode_plus.<locals>.get_input_ids.<locals>.<genexpr>z\Input is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers.r�r�r�r#r$r�Ts307308(�309310(�z=PreTrainedTokenizer._batch_encode_plus.<locals>.get_input_idsz�return_offset_mapping is not available when using Python tokenizers. To use this feature, change your tokenizer to one deriving from transformers.PreTrainedTokenizerFast.r)r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�)r�r�rDr&r2�_batch_prepare_for_modelr)r!rr�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r�Z	input_idsZids_or_pair_idsr�rrr�
batch_outputsr#r�r$�_batch_encode_plus7sL������z&PreTrainedTokenizer._batch_encode_plus�batch_ids_pairscCs�i}|D]W\}}|j||fid|�dtjj�d|j�d|�d|�dd�dd�d	d311�d|312�d|�d
|
�d|�dd�dd313�d|�d|��}|��D]\}}||vrSg||<||�|�qGq|j||j||||d�}t||	d�}|S)a�314        Prepares a sequence of input id, or a pair of sequences of inputs ids so that it can be used by the model. It315        adds special tokens, truncates sequences if overflowing while taking into account the special tokens and316        manages a moving window (with user defined stride) for overflowing tokens317 318        Args:319            batch_ids_pairs: list of tokenized input ids or input ids pairs320        r�rrr�r�r�Nr�r�Fr�r�r�r�r�rr�r�)rr�r�r�r�)Ztensor_type)rr�321DO_NOT_PADr�r0r2�padr)r!rr�r�r�r�r�r�r�r�r�r�r�r�r�r�r�r322rrZoutputsr�r�r#r#r$r	�sj�������	�323���
��������	z,PreTrainedTokenizer._batch_prepare_for_modelcKs||fS)a�324        Performs any necessary transformations before tokenization.325 326        This method should pop the arguments from kwargs and return the remaining `kwargs` as well. We test the327        `kwargs` at the end of the encoding process to be sure all the arguments have been used.328 329        Args:330            text (`str`):331                The text to prepare.332            is_split_into_words (`bool`, *optional*, defaults to `False`):333                Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the334                tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace)335                which it will tokenize. This is useful for NER or token classification.336            kwargs (`dict[str, Any]`, *optional*):337                Keyword arguments to use for the tokenization.338 339        Returns:340            `tuple[str, dict[str, Any]]`: The prepared text and the unused kwargs.341        r#)r!r-r�r�r#r#r$r��sz,PreTrainedTokenizer.prepare_for_tokenizationr�r��already_has_special_tokenscsD|r|dur342td��t�j||dd�Sdg|rt|�ndt|�S)a�343        Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding344        special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods.345 346        Args:347            token_ids_0 (`list[int]`):348                List of ids of the first sequence.349            token_ids_1 (`list[int]`, *optional*):350                List of ids of the second sequence.351            already_has_special_tokens (`bool`, *optional*, defaults to `False`):352                Whether or not the token list is already formatted with special tokens for the model.353 354        Returns:355            A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.356        NzYou should not supply a second sequence if the provided sequence of ids is already formatted with special tokens for the model.T)r�r�rr)r�rG�get_special_tokens_maskr1)r!r�r�rrHr#r$r�s��z+PreTrainedTokenizer.get_special_tokens_maskr��skip_special_tokenscC�dSrFr#�r!r�rr#r#r$�convert_ids_to_tokensr�z)PreTrainedTokenizer.convert_ids_to_tokenscCrrFr#rr#r#r$rr�cCs�t|t�r||jvr|j|jS|�|�Sg}|D]%}t|�}|r'||jvr'q||jvr6|�|j|j�q|�|�|��q|S)a�357        Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and358        added tokens.359 360        Args:361            ids (`int` or `list[int]`):362                The token id (or token ids) to convert to tokens.363            skip_special_tokens (`bool`, *optional*, defaults to `False`):364                Whether or not to remove special tokens in the decoding.365 366        Returns:367            `str` or `list[str]`: The decoded token(s).368        )r�r�r�r��_convert_id_to_tokenZall_special_idsr2)r!r�rr>r�r#r#r$rs369370371372r�cCr�rFr�)r!r�r#r#r$r4r�z(PreTrainedTokenizer._convert_id_to_tokencCs373d�|�S)NrZ)r�)r!r>r#r#r$�convert_tokens_to_string7s374z,PreTrainedTokenizer.convert_tokens_to_string�	token_ids�clean_up_tokenization_spaces�spaces_between_special_tokenscs|�dd��_�j||d�}t|t�r|g}t�j���t�j��fdd��j	D�B}g}g}	|D]-}375|r<|376�jvr<q2|377|vrZ|	rT��378|	�}t|�dkrR|�|�g}	|�|379�q2|	�|380�q2|	rj|���381|	��|rrd�
|�}nd�
|�}|dur}|n�j}|r���|�}
|
S|S)	NZuse_source_tokenizerF)rcs h|]}��|��jkr|�qSr#)r�r�rLr�r#r$�	<setcomp>Isz.PreTrainedTokenizer._decode.<locals>.<setcomp>rrZr)r�r�rr�rCrr�r�r�r�rr1r2r�rZclean_up_tokenization)r!rrrrr�Zfiltered_tokensZlegacy_added_tokensZ	sub_textsZcurrent_sub_textr(�stringr-Z382clean_textr#r�r$�_decode:sB383 �384385386��387zPreTrainedTokenizer._decode)FrFr�)FNT):r?r@rArBr%�propertyrvr�r�r�rXrCr�rr��setterrr�r�r�rDr�rr�r�rr�r�r�r�r�rr
rZDO_NOT_TRUNCATErr
rrrrrrrrrr	r&r	rr�rrrrrrrYr#r#rHr$r��s�	,(NP*	��������	�388���
�������389�Z����390���
��������������391�[��������	�392���
������F���393������"���394� ������r�)0rBr|r�r�r_�collectionsr�typingrrrrZtokenization_utils_baserr	r395rrr
rrrrrrr�utilsrrrrZ396get_loggerr?r<ZSPECIAL_TOKENS_MAP_FILEZADDED_TOKENS_FILEZTOKENIZER_CONFIG_FILErrErcrfrrrwryrDrCrr�r#r#r#r$�<module>s0
<397h;