hymenjj/llama-cpp-python-prebuilt
0
1�
2 00�i$ � � � d dl mZ d dlZd dlmZmZmZ d dlZd dlmZ G d� dej � � Z3 G d� de4� � Z G d � d5e6� � Z dS ) � )� annotationsN)�List�Optional�Any)r c �\ � e Zd Zej ddd �� � Zej ddd�� � Zd7S )�BaseLlamaTokenizerT�text�bytes�add_bos�bool�special�return� List[int]c � � t �)z�Tokenize the text into tokens.8 9 Args:10 text: The utf-8 encoded string to tokenize.11 add_bos: Whether to add a beginning of sequence token.12 special: Whether to tokenize special tokens.13 ��NotImplementedError��selfr r r
s �D/usr/local/lib/python3.11/site-packages/llama_cpp/llama_tokenizer.py�tokenizezBaseLlamaTokenizer.tokenize s14 � � "�!� NF�tokens� prev_tokens�Optional[List[int]]c � � t �)a
Detokenize the tokens into text.15 16 Args:17 tokens: The list of tokens to detokenize.18 prev_tokens: The list of previous tokens. Offset mapping will be performed if provided.19 special: Whether to detokenize special tokens.20 r �r r r r
s r �21detokenizez BaseLlamaTokenizer.detokenize s22 � � "�!r �TT�r r23 r r r
r r r �NF�r r r r r
r r r24 )�__name__�25__module__� __qualname__�abc�abstractmethodr r � r r r r so � � � � � ���AE�26"� 27"� 28"� 29"� ��30"� �� ,0� �
"�
"�
"�
"� ��
"�
"�
"r r c �\ � e Zd Zdd�Z ddd �Z d d d�Z dd d�Zd d�Zed d�� � Z d
S )!�LlamaTokenizer�llama�llama_cpp.Llamac � � |j | _ d S �N)�_model)r r* s r �__init__zLlamaTokenizer.__init__. s � ��l�� � � r Tr r31 r r r
r r c �< � | j � |||�� � S )N�r r
)r. r r s r r zLlamaTokenizer.tokenize1 s! � � �{�#�#�D�'�7�#�K�K�Kr NFr r r c �: � | j � ||�� � S )N)r
)r. r r s r r zLlamaTokenizer.detokenize6 s � � �{�%�%�f�g�%�>�>�>r �strc �\ � | � |� dd�� � ||�� � S )N�utf-8�ignore��errorsr1 )r �encoder s r r9 zLlamaTokenizer.encode> s6 � � �}�}� �K�K���K� 1� 1�7�G� � 32� 33� 34r c �V � | � |� � � dd�� � S )Nr5 r6 r7 )r �decode)r r s r r; zLlamaTokenizer.decodeE s' � ����v�&�&�-�-�g�h�-�G�G�Gr �path�'LlamaTokenizer'c �@ � | t j |d�� � � � S )NT)�35model_path�36vocab_only)� llama_cpp�Llama)�clsr< s r �from_ggml_filez LlamaTokenizer.from_ggml_fileH s! � ��s�9�?�d�t�D�D�D�E�E�Er )r* r+ r r r r! )r r3 r r r
r r r )r r r r3 )r< r3 r r= )37r" r# r$ r/ r r r9 r; � classmethodrD r' r r r) r) - s� � � � � � �#� #� #� #� BF�L� L� L� L� L� ,0� � ?� ?� ?� ?� ?� @D�38� 39� 40� 41� 42�H� H� H� H� �F� F� F� �[�F� F� Fr r) c �H � e Zd Zdd�Z ddd �Z ddd�Zed d�� � Zd
S ) �LlamaHFTokenizer� hf_tokenizerr c � � || _ d S r- )rH )r rH s r r/ zLlamaHFTokenizer.__init__N s � � (����r Tr r43 r r r
r r c �d � | j � |� dd�� � |�� � S )Nr5 r6 r7 )�add_special_tokens)rH r9 r; r s r r zLlamaHFTokenizer.tokenizeQ s9 � � � �'�'� �K�K���K� 1� 1�g� (� 44� 45� 46r NFr r r c �f � | }|�|| j � ||z |�� � � dd�� � }| j � ||�� � � dd�� � }|t |� � d � S | j � ||�� � � dd�� � S )N)�skip_special_tokensr5 r6 r7 )rH r; r9 �len)r r r r
rM r � prev_texts r r zLlamaHFTokenizer.detokenizeX s� � � #*�k�� � "��$�+�+��f�$�:M� ,� � ��f�W�X�f�.�.�
� �)�0�0��1D� 1� � ��f�W�X�f�.�.�
� ��I���(�(�)� )��$�+�+��,?� ,� � ��f�W�X�f�.�.�
/r � pretrained_model_name_or_pathr3 �'LlamaHFTokenizer'c � � ddl m} n # t $ r t d� � �w xY w|� |�� � } | |� � S )Nr )�
AutoTokenizerzsThe `transformers` library is required to use the `HFTokenizer`.You can install it with `pip install transformers`.)rP )� transformersrS � ImportError�from_pretrained)rC rP rS rH s r rV z LlamaHFTokenizer.from_pretrainedl s� � � � 2� 2� 2� 2� 2� 2� 2��� � � � �F�� �
� ����47 %�4�4�*G� 5� 48� 49� � �s�<� � � s � �#)rH r r r r r! )rP r3 r rQ )r" r# r$ r/ r r rE rV r' r r rG rG M s� � � � � � �)� )� )� )� BF�50� 51� 52� 53� 54� ,0� � /� /� /� /� /�( � !� !� !� �[� !� !� !r rG )
�55__future__r r% �typingr r r rA �llama_cpp.llama_types�ABCr r) rG r' r r �<module>r[ s �� "� "� "� "� "� "� 56�57�58�59�� � � � � � � � � � � � � � &� &� &� &� &� &� "� "� "� "� "��� "� "� "�> F� F� F� F� F�'� F� F� F�@+!� +!� +!� +!� +!�)� +!� +!� +!� +!� +!r