Apppublic

XaviXva/Video-LLaVA

sourceHugging Faceapache-2.0updated 3y agoView on Hugging Face

0likes

rootllavamodelmultimodal_encoderlanguagebindvideo__pycache__tokenization_video.cpython-310.pyc

tokenization_video.cpython-310.pyc38 linesDownload Raw Back to __pycache__

1o

2��e�3�@sbddlmZddlmZe�e�Zddd�Zddiddid�Zdd	iZ	diiZ4Gd5d�de�ZdS)
�)�
CLIPTokenizer)�loggingz6vocab.jsonz7merges.txt)�8vocab_file�merges_filezlb203/LanguageBind-VideozGhttps://huggingface.co/lb203/LanguageBind-Video/resolve/main/vocab.jsonzGhttps://huggingface.co/lb203/LanguageBind-Video/resolve/main/merges.txt�Mcs@eZdZdZeZeZeZ	ddgZ9					d	�fdd�	Z�ZS)10�LanguageBindVideoTokenizera<11    Construct a CLIP tokenizer. Based on byte-level Byte-Pair-Encoding.12 13    This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to14    this superclass for more information regarding those methods.15 16    Args:17        vocab_file (`str`):18            Path to the vocabulary file.19        merges_file (`str`):20            Path to the merges file.21        errors (`str`, *optional*, defaults to `"replace"`):22            Paradigm to follow when decoding bytes to UTF-8. See23            [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information.24        unk_token (`str`, *optional*, defaults to `<|endoftext|>`):25            The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this26            token instead.27        bos_token (`str`, *optional*, defaults to `<|startoftext|>`):28            The beginning of sequence token.29        eos_token (`str`, *optional*, defaults to `<|endoftext|>`):30            The end of sequence token.31    �	input_ids�attention_mask�replace�
<|endoftext|>�<|startoftext|>c	s(tt|�j|||||||fi|��dS)N)�superr�__init__)	�selfrr�errors�	unk_token�	bos_token�	eos_token�	pad_token�kwargs��	__class__��PD:\LLaVA\llava\model\multimodal_encoder\languagebind\video\tokenization_video.pyr:s32�33�z#LanguageBindVideoTokenizer.__init__)r34rrrr)
�__name__�35__module__�__qualname__�__doc__�VOCAB_FILES_NAMES�vocab_files_names�PRETRAINED_VOCAB_FILES_MAP�pretrained_vocab_files_map�&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES�max_model_input_sizes�model_input_namesr�
__classcell__rrrrrs�rN)�transformersr�transformers.utilsr�36get_loggerr�loggerrr r"�PRETRAINED_INIT_CONFIGURATIONrrrrr�<module>s 37����38��