Aluode/PerceptionLabPortable
0
1o
2 .�Yi; � @ s� d Z ddlZddlZddlmZ ddlmZmZ ddlZ ddl3m Z m Z ddl
mZmZ ddlmZ e�e�Zd d4iZed d �G d
d� de ��ZdgZdS )zTokenization classes.� N)�copyfile)�Any�Optional� )�5AddedToken�PreTrainedTokenizer)�SPIECE_UNDERLINE�logging)�requires�6vocab_filez spiece.model)�
sentencepiece)�backendsc sd e Zd ZdZeZddddddddd d7d d gd
f deeee f dd
f� fdd�
Z8e dd� �Z dd� Z
dd� Zdd� Zdd� Zd edee fd d �Zd d � Zd!d"� Zd#d$� Z
d4d%ee d&eee dee fd'd(�Z d5d%ee d&eee d)edee f� fd*d+�
Z
d4d%ee d&eee dee fd,d-�Zd4d.ed/ee dee fd0d1�Z � fd2d3�Z � Z S )6� CpmTokenizerzPRuns pre-tokenization with Jieba-RS segmentation tool. It is used in CPM models.FTz<s>z</s>z<unk>z<sep>z<pad>z<cls>z<mask>z<eop>z<eod>N�sp_model_kwargs�returnc s� t | t�r t| ddd�n| } |
du ri n|
| _|| _|| _|| _|| _tj d i | j��| _9| j10� |� zddl }W n t
yJ } z|�d��d}~ww || _t�dd�| _t� jd |||||||| |11| | | jd � |�� d12| _dS ) a�
13 Construct a CPM tokenizer. Based on [Jieba-RS](https://pypi.org/project/rjieba/) and14 [SentencePiece](https://github.com/google/sentencepiece).15 16 This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should17 refer to this superclass for more information regarding those methods.18 19 Args:20 vocab_file (`str`):21 [SentencePiece](https://github.com/google/sentencepiece) file (generally has a .spm extension) that22 contains the vocabulary necessary to instantiate a tokenizer.23 do_lower_case (`bool`, *optional*, defaults to `True`):24 Whether to lowercase the input when tokenizing.25 remove_space (`bool`, *optional*, defaults to `True`):26 Whether to strip the text when tokenizing (removing excess spaces before and after the string).27 keep_accents (`bool`, *optional*, defaults to `False`):28 Whether to keep accents when tokenizing.29 bos_token (`str`, *optional*, defaults to `"<s>"`):30 The beginning of sequence token that was used during pretraining. Can be used a sequence classifier31 token.32 33 <Tip>34 35 When building a sequence using special tokens, this is not the token that is used for the beginning of36 sequence. The token used is the `cls_token`.37 38 </Tip>39 40 eos_token (`str`, *optional*, defaults to `"</s>"`):41 The end of sequence token.42 43 <Tip>44 45 When building a sequence using special tokens, this is not the token that is used for the end of46 sequence. The token used is the `sep_token`.47 48 </Tip>49 50 unk_token (`str`, *optional*, defaults to `"<unk>"`):51 The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be52 this token instead.53 sep_token (`str`, *optional*, defaults to `"<sep>"`):54 The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences55 for sequence classification or for a text and a question for question answering. It is also used as the56 last token of a sequence built with special tokens.57 pad_token (`str`, *optional*, defaults to `"<pad>"`):58 The token used for padding, for example when batching sequences of different lengths.59 cls_token (`str`, *optional*, defaults to `"<cls>"`):60 The classifier token which is used when doing sequence classification (classification of the whole61 sequence instead of per-token classification). It is the first token of the sequence when built with62 special tokens.63 mask_token (`str`, *optional*, defaults to `"<mask>"`):64 The token used for masking values. This is the token used when training this model with masked language65 modeling. This is the token which the model will try to predict.66 additional_special_tokens (`list[str]`, *optional*, defaults to `["<eop>", "<eod>"]`):67 Additional special tokens used by the tokenizer.68 69 Attributes:70 sp_model (`SentencePieceProcessor`):71 The *SentencePiece* processor that is used for every conversion (string, tokens and IDs).72 TF)�lstrip�rstripNr zzYou need to install rjieba to use CpmTokenizer or CpmTokenizerFast. See https://pypi.org/project/rjieba/ for installation.z 73u ▂▃) �
do_lower_case� remove_space� keep_accents� bos_token� eos_token� unk_token� sep_token� pad_token� cls_token�74mask_token�additional_special_tokensr r � )�75isinstance�strr r r r r r �spm�SentencePieceProcessor�sp_model�Load�rjieba�ModuleNotFoundError� __class__Zjieba� maketransZ76translator�super�__init__Z_pad_token_type_id)�selfr r r r r r r r r r r r r �kwargsr% �error�r' r ��E:\DocsHouse\542 percep lab latest\PerceptionLab\PerceptionLab_Portable\python_embed\Lib\site-packages\transformers/models/cpm/tokenization_cpm.pyr* ( sF O ����
�77zCpmTokenizer.__init__c C s78 t | j�S �N)�lenr# �r+ r r r/ �79vocab_size� s 80zCpmTokenizer.vocab_sizec s( � fdd�t � j�D �}|�� j� |S )Nc s i | ]}� � |�|�qS r )Zconvert_ids_to_tokens)�.0�ir2 r r/ �81<dictcomp>� s z*CpmTokenizer.get_vocab.<locals>.<dictcomp>)�ranger3 �updateZadded_tokens_encoder)r+ Zvocabr r2 r/ � get_vocab� s zCpmTokenizer.get_vocabc C s | j �� }d |d< |S )Nr# )�__dict__�copy)r+ �stater r r/ � __getstate__� s 82zCpmTokenizer.__getstate__c C s< || _ t| d�s i | _tjdi | j��| _| j�| j� d S )Nr r )r: �hasattrr r! r" r# r$ r )r+ �dr r r/ � __setstate__� s83 84zCpmTokenizer.__setstate__c C sj | j r
d�|�� �� �}n|}|�dd��dd�}| js,t�d|�}d�dd� |D ��}| jr3|� � }|S ) N� z``�"z''ZNFKD� c S s g | ] }t �|�s|�qS r )� unicodedata� combining)r4 �cr r r/ �85<listcomp>� s z0CpmTokenizer.preprocess_text.<locals>.<listcomp>)86r �join�strip�split�replacer rD � normalizer �lower)r+ ZinputsZoutputsr r r/ �preprocess_text� s z CpmTokenizer.preprocess_text�textc C s� | � |�}| jj|td�}g }|D ][}t|�dkrg|d dkrg|d �� rg| j�|dd� �td��}|d tkrZ|d d tkrZt|d �dkrP|dd� }n87|d dd� |d< |� |d � |�88|� q|� |� q|S ) zTokenize a string.)Zout_type� ������,�����NrC r ) rN r# �encoder r1 �isdigitZEncodeAsPiecesrK r �append�extend)r+ rO �piecesZ89new_piecesZpieceZ90cur_piecesr r r/ � _tokenize� s 91$ zCpmTokenizer._tokenizec C � | j �|�S )z0Converts a token (str) in an id using the vocab.)r# Z PieceToId)r+ �tokenr r r/ �_convert_token_to_id� � z!CpmTokenizer._convert_token_to_idc C rZ )z=Converts an index (integer) in a token (str) using the vocab.)r# Z IdToPiece)r+ �indexr r r/ �_convert_id_to_token� r] z!CpmTokenizer._convert_id_to_tokenc C s d� |��td��� }|S )zIConverts a sequence of tokens (strings for sub-words) in a single string.rC rA )rH rK r rI )r+ �tokensZ92out_stringr r r/ �convert_tokens_to_string� s z%CpmTokenizer.convert_tokens_to_string� token_ids_0� token_ids_1c C s8 | j g}| jg}|du r|| | S || | | | S )a� 93 Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and94 adding special tokens. An XLNet sequence has the following format:95 96 - single sequence: `X <sep> <cls>`97 - pair of sequences: `A <sep> B <sep> <cls>`98 99 Args:100 token_ids_0 (`list[int]`):101 List of IDs to which the special tokens will be added.102 token_ids_1 (`list[int]`, *optional*):103 Optional second list of IDs for sequence pairs.104 105 Returns:106 `list[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.107 N)� sep_token_idZ cls_token_id)r+ rb rc �sep�clsr r r/ � build_inputs_with_special_tokens� s108 z-CpmTokenizer.build_inputs_with_special_tokens�already_has_special_tokensc s^ |r t � j||dd�S |dur$dgt|� dg dgt|� ddg S dgt|� ddg S )a� 109 Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding110 special tokens using the tokenizer `prepare_for_model` method.111 112 Args:113 token_ids_0 (`list[int]`):114 List of IDs.115 token_ids_1 (`list[int]`, *optional*):116 Optional second list of IDs for sequence pairs.117 already_has_special_tokens (`bool`, *optional*, defaults to `False`):118 Whether or not the token list is already formatted with special tokens for the model.119 120 Returns:121 `list[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.122 T)rb rc rh Nr rP )r) �get_special_tokens_maskr1 )r+ rb rc rh r. r r/ ri s �*z$CpmTokenizer.get_special_tokens_maskc C sT | j g}dg}|du rt|| �dg | S t|| �dg t|| �dg | S )a� 123 Create a mask from the two sequences passed to be used in a sequence-pair classification task. An XLNet124 sequence pair mask has the following format:125 126 ```127 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1128 | first sequence | second sequence |129 ```130 131 If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s).132 133 Args:134 token_ids_0 (`list[int]`):135 List of IDs.136 token_ids_1 (`list[int]`, *optional*):137 Optional second list of IDs for sequence pairs.138 139 Returns:140 `list[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s).141 � Nr rP )rd r1 )r+ rb rc re Zcls_segment_idr r r/ �$create_token_type_ids_from_sequences( s142 (z1CpmTokenizer.create_token_type_ids_from_sequences�save_directory�filename_prefixc C s� t j�|�st�d|� d�� d S t j�||r|d ndtd �}t j�| j�t j�|�kr?t j� | j�r?t143| j|� |fS t j� | j�sgt |d��}| j �
� }|�|� W d � |fS 1 sbw Y |fS )NzVocabulary path (z) should be a directory�-rC r �wb)�os�path�isdir�loggerr- rH �VOCAB_FILES_NAMES�abspathr �isfiler �openr# Zserialized_model_proto�write)r+ rl rm Zout_vocab_file�fiZcontent_spiece_modelr r r/ �save_vocabularyG s" �( � 144 145��z CpmTokenizer.save_vocabularyc s2 t � j|i |��}|�dd��dd��dd�}|S )NrA rC u ▂u ▃�146)r) �_decoderK )r+ �argsr, rO r. r r/ r| X s zCpmTokenizer._decoder0 )NF) �__name__�147__module__� __qualname__�__doc__rt Zvocab_files_namesr �dictr r r* �propertyr3 r9 r= r@ rN �listrY r\ r_ ra �intrg �boolri rk �tuplerz r| �
__classcell__r r r. r/ r " st ���w148 ��149�150���151��� ��152�153� r )r� rp rD �shutilr �typingr r r r! Ztokenization_utilsr r �utilsr r Zutils.import_utilsr154 Z155get_loggerr~ rs rt r �__all__r r r r/ �<module> s 156 157=