Aluode/PerceptionLabPortable

1o

2.�YitT
�@s�dZd
dl
Z
d
dlZd
dlZd
dlZd
dlZd
dlZd
dlZd
dlZd
dl	m3Z4
d
dlmZ
d
dlm
Z
mZmZmZmZ
d
dlZd
dlZd
dlmZ
ddlmZmZ
dd	lmZ
dd5lmZ
ddlmZm Z m!Z!
ddl"m#Z#
dd
l$m%Z%m&Z&
e!�rddlm'Z'
ddl(m)Z)m*Z*m+Z+m,Z,m-Z-
ddl.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;m<Z<m=Z=m>Z>
ddl?m@Z@
e<�r�ddlAmBZB
e>�CeD�
ZEeddd�ZFe8eeG�
jH�
ZIddddd�ZJejKdkr�ejLZLnejLZLGdd�dedd�ZMGd d!�d!edd�ZNGd"d#�d#edd�ZOGd$d%�d%edd�ZPGd&d'�d'edd�ZQGd(d)�d)edd�ZRGd*d+�d+edd�ZSGd,d-�d-edd�ZTGd.d/�d/eTeSdd�ZUGd0d1�d1edd�ZVe6Gd2d3�d3��
ZWGd4d�de4�ZXe7eXjY�
eX_YeXjYjdu
�
rveXjYjjZd5d6d7d8�eXjY_dSdS)9z87Processing saving/loading class for common processors.8�N)
�	dataclass)
�Path)�Any�Optional�	TypedDict�TypeVar�Union)
�EntryNotFoundError�
)�9AudioInput�10load_audio)
�custom_object_save)
�BatchFeature)�ChannelDimension�11ImageInput�is_vision_available)
�render_jinja_template)�12VideoInput�
VideoMetadata)
�PILImageResampling)�PaddingStrategy�PreTokenizedInput�PreTrainedTokenizerBase�	TextInput�TruncationStrategy)�AUDIO_TOKENIZER_NAME�CHAT_TEMPLATE_DIR�CHAT_TEMPLATE_FILE�#LEGACY_PROCESSOR_CHAT_TEMPLATE_FILE�PROCESSOR_NAME�PushToHubMixin�13TensorType�cached_file�	copy_func�direct_transformers_import�download_url�is_offline_mode�
is_remote_url�is_torch_available�list_repo_templates�logging)
�deprecate_kwarg)
�PreTrainedAudioTokenizerBase�SpecificProcessorType�ProcessorMixin)
�boundrZFeatureExtractionMixinZImageProcessingMixinZBaseVideoProcessor)Z
AutoTokenizerZAutoFeatureExtractorZAutoImageProcessorZAutoVideoProcessor)��c@s:
eZ
dZUd
Zeeeeeeeefe	d<eeeeeeefe	d<eeeeeeeefe	d<ee14e	d<ee15eefe	d<ee16ee
fe	d<eee	d<eee	d	<ee17e	d18<eee	d<ee19e	d<ee20e	d
<ee21e	d<ee22e	d<ee23e	d<ee24e	d<ee25e	d<eee	d<ee26e	d<dS)�27TextKwargsaL28    Keyword arguments for text processing. For extended documentation, check out tokenization_utils_base methods and29    docstrings associated.30 31    Attributes:32        add_special_tokens (`bool`, *optional*)33            Whether or not to add special tokens when encoding the sequences.34        padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*)35            Activates and controls padding.36        truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*):37            Activates and controls truncation.38        max_length (`int`, *optional*):39            Controls the maximum length to use by one of the truncation/padding parameters.40        stride (`int`, *optional*):41            If set, the overflowing tokens will contain some tokens from the end of the truncated sequence.42        is_split_into_words (`bool`, *optional*):43            Whether or not the input is already pre-tokenized.44        pad_to_multiple_of (`int`, *optional*):45            If set, will pad the sequence to a multiple of the provided value.46        return_token_type_ids (`bool`, *optional*):47            Whether to return token type IDs.48        return_attention_mask (`bool`, *optional*):49            Whether to return the attention mask.50        return_overflowing_tokens (`bool`, *optional*):51            Whether or not to return overflowing token sequences.52        return_special_tokens_mask (`bool`, *optional*):53            Whether or not to return special tokens mask information.54        return_offsets_mapping (`bool`, *optional*):55            Whether or not to return `(char_start, char_end)` for each token.56        return_length (`bool`, *optional*):57            Whether or not to return the lengths of the encoded inputs.58        verbose (`bool`, *optional*):59            Whether or not to print more information and warnings.60        padding_side (`str`, *optional*):61            The side on which padding will be applied.62        return_mm_token_type_ids (`bool`, *optional*):63            Whether to return multimodal token type ids indicating mm placeholder token positions.64    Z	text_pairZtext_targetZtext_pair_target�add_special_tokens�padding�65truncation�66max_lengthZstrideZis_split_into_words�pad_to_multiple_ofZreturn_token_type_ids�return_attention_maskZreturn_overflowing_tokensZreturn_special_tokens_mask�return_offsets_mappingZ
return_length�verboseZpadding_sideZreturn_mm_token_type_idsN)�__name__�67__module__�__qualname__�__doc__rrrr�list�__annotations__�bool�strrr�int�rDrD��E:\DocsHouse\542 percep lab latest\PerceptionLab\PerceptionLab_Portable\python_embed\Lib\site-packages\transformers/processing_utils.pyr2ds*68
 '
 
















r2F)
�totalc@s
eZ
dZUd
Zeeed<eeee	fed<eeee	fed<ee69de	fed<eeed<eeed<eeed	<ee70eeefed71<ee72eeefed<eeed<eeee	fed
<eeed<ee
ed<ee73ee
fed<eeed<dS)�ImagesKwargsah74    Keyword arguments for image processing. For extended documentation, check the appropriate ImageProcessor75    class methods and docstrings.76 77    Attributes:78        do_resize (`bool`, *optional*):79            Whether to resize the image.80        size (`dict[str, int]`, *optional*):81            Resize the shorter side of the input to `size["shortest_edge"]`.82        crop_size (`dict[str, int]`, *optional*):83            Desired output size when applying center-cropping.84        resample (`PILImageResampling`, *optional*):85            Resampling filter to use if resizing the image.86        do_rescale (`bool`, *optional*):87            Whether to rescale the image by the specified scale `rescale_factor`.88        rescale_factor (`int` or `float`, *optional*):89            Scale factor to use if rescaling the image.90        do_normalize (`bool`, *optional*):91            Whether to normalize the image.92        image_mean (`float` or `list[float]`, *optional*):93            Mean to use if normalizing the image.94        image_std (`float` or `list[float]`, *optional*):95            Standard deviation to use if normalizing the image.96        do_pad (`bool`, *optional*):97            Whether to pad the image to the `(max_height, max_width)` of the images in the batch.98        pad_size (`dict[str, int]`, *optional*):99            The size `{"height": int, "width" int}` to pad the images to.100        do_center_crop (`bool`, *optional*):101            Whether to center crop the image.102        data_format (`ChannelDimension` or `str`, *optional*):103            The channel dimension format for the output image.104        input_data_format (`ChannelDimension` or `str`, *optional*):105            The channel dimension format for the input image.106        device (`str`, *optional*):107            The device to use for processing (e.g. "cpu", "cuda"), only relevant for fast image processing.108    �	do_resize�size�	crop_sizer�resample�109do_rescale�rescale_factor�do_normalize�110image_mean�	image_stdZdo_padZpad_size�do_center_crop�data_format�input_data_format�deviceN)r;r<r=r>rrAr@�dictrBrCr�floatr?rrDrDrDrErG�s"111
%













rGc@sB
eZ
dZUd
Zeeed<eeed<eeee	fed<eeed<eded<eeed<ee112ed	<eeed113<eee114ee115fed<eee116ee117fed<eeed
<eeee	fed<ee
ed<eeee
fed<eeed<eeed<eeeefed<eee	e118fed<ee	ed<eeed<dS)�VideosKwargsa	119    Keyword arguments for video processing.120 121    Attributes:122        do_convert_rgb (`bool`):123            Whether to convert the video to RGB format.124        do_resize (`bool`):125            Whether to resize the video.126        size (`dict[str, int]`, *optional*):127            Resize the shorter side of the input to `size["shortest_edge"]`.128        default_to_square (`bool`, *optional*, defaults to `self.default_to_square`):129            Whether to default to a square when resizing, if size is an int.130        resample (`PILImageResampling`, *optional*):131            Resampling filter to use if resizing the video.132        do_rescale (`bool`, *optional*):133            Whether to rescale the video by the specified scale `rescale_factor`.134        rescale_factor (`int` or `float`, *optional*):135            Scale factor to use if rescaling the video.136        do_normalize (`bool`, *optional*):137            Whether to normalize the video.138        image_mean (`float` or `list[float]`, *optional*):139            Mean to use if normalizing the video.140        image_std (`float` or `list[float]`, *optional*):141            Standard deviation to use if normalizing the video.142        do_center_crop (`bool`, *optional*):143            Whether to center crop the video.144        do_sample_frames (`bool`, *optional*):145            Whether to sample frames from the video before processing or to process the whole video.146        video_metadata (`Union[VideoMetadata, dict]`, *optional*):147            Metadata of the video containing information about total duration, fps and total number of frames.148        num_frames (`int`, *optional*):149            Maximum number of frames to sample when `do_sample_frames=True`.150        fps (`int` or `float`, *optional*):151            Target frames to sample per second when `do_sample_frames=True`.152        crop_size (`dict[str, int]`, *optional*):153            Desired output size when applying center-cropping.154        data_format (`ChannelDimension` or `str`, *optional*):155            The channel dimension format for the output video.156        input_data_format (`ChannelDimension` or `str`, *optional*):157            The channel dimension format for the input video.158        return_metadata (`ChannelDimension` or `str`, *optional*):159            Whether to return video metadata or not.160    Zdo_convert_rgbrHrIZdefault_to_squarerrKrLrMrNrOrPrQrJrRrSrT�do_sample_framesZvideo_metadata�fps�161num_framesZreturn_metadataN)r;r<r=r>rrAr@rUrBrCrVrr?rrrDrDrDrErW�s,162
,


















rWc@s�eZ
dZUd
Zeeed<eeej	e163ee164ej	e165e166efed<eeee
efed<eeed<eeed<eeed<eeed<d	S)167�AudioKwargsa�168    Keyword arguments for audio processing.169 170    Attributes:171        sampling_rate (`int`, *optional*):172            The sampling rate at which the `raw_speech` input was sampled.173        raw_speech (`np.ndarray`, `list[float]`, `list[np.ndarray]`, `list[list[float]]`):174            The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float175            values, a list of numpy arrays or a list of list of float values. Must be mono channel audio, not176            stereo, i.e. single float per timestep.177        padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*):178            Select a strategy to pad the returned sequences (according to the model's padding side and padding179            index) among:180 181            - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single182                sequence if provided).183            - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum184                acceptable input length for the model if that argument is not provided.185            - `False` or `'do_not_pad'`186        max_length (`int`, *optional*):187            Maximum length of the returned list and optionally padding length (see above).188        truncation (`bool`, *optional*):189            Activates truncation to cut input sequences longer than *max_length* to *max_length*.190        pad_to_multiple_of (`int`, *optional*):191            If set, will pad the sequence to a multiple of the provided value.192        return_attention_mask (`bool`, *optional*):193            Whether or not [`~ASTFeatureExtractor.__call__`] should return `attention_mask`.194    �
sampling_rateZ195raw_speechr4r6r5r7r8N)r;r<r=r>rrCr@r�np�ndarrayr?rVrArBrrDrDrDrEr[
s196
,





r[c@s"eZ
dZUeeeefed
<dS)�CommonKwargs�return_tensorsN)r;r<r=rrrBr!r@rDrDrDrEr_B
s197
r_c@speZ
dZUd
ZiZiej�
Zeed<iej�
Z	eed<ie198j�
Ze199ed<iej�
Z
eed<iej�
Zeed<dS)�ProcessingKwargsa�200    Base class for kwargs passing to processors.201    In case a model has specific kwargs that are not present in the base class or default values for existing keys,202    it should have its own `ModelProcessorKwargs` class that inherits from `ProcessingKwargs` to provide:203        1) Additional typed keys and that this model requires to process inputs.204        2) Default values for existing keys under a `_defaults` attribute.205    New keys have to be defined as follows to ensure type hinting is done correctly.206 207    ```python208    # adding a new image kwarg for this model209    class ModelImagesKwargs(ImagesKwargs, total=False):210        new_image_kwarg: Optional[bool]211 212    class ModelProcessorKwargs(ProcessingKwargs, total=False):213        images_kwargs: ModelImagesKwargs214        _defaults = {215            "images_kwargs: {216                "new_image_kwarg": False,217            }218            "text_kwargs": {219                "padding": "max_length",220            },221        }222 223    ```224 225    For Python 3.8 compatibility, when inheriting from this class and overriding one of the kwargs,226    you need to manually update the __annotations__ dictionary. This can be done as follows:227 228    ```python229    class CustomProcessorKwargs(ProcessingKwargs, total=False):230        images_kwargs: CustomImagesKwargs231 232    CustomProcessorKwargs.__annotations__["images_kwargs"] = CustomImagesKwargs  # python 3.8 compatibility233    ```python234 235    �
common_kwargs�text_kwargs�
images_kwargs�
videos_kwargs�audio_kwargsN)r;r<r=r>�	_defaultsr_r@rbr2rcrGrdrWrer[rfrDrDrDrEraF
s$236
&
�
�
�
�
�rac@sreZ
dZUd
ZdZeeeed<dZ	eeee237e238fed<dZeeed<dZ
eeed<dZeeed<dS)	�TokenizerChatTemplateKwargsaU	239    Keyword arguments for tokenizer's `apply_chat_template`, when it is called from within a processor.240 241    tools (`list[Dict]`, *optional*):242        A list of tools (callable functions) that will be accessible to the model. If the template does not243        support function calling, this argument will have no effect. Each tool should be passed as a JSON Schema,244        giving the name, description and argument types for the tool. See our245        [chat templating guide](https://huggingface.co/docs/transformers/main/en/chat_templating#automated-function-conversion-for-tool-use)246        for more information.247    documents (`list[dict[str, str]]`, *optional*):248        A list of dicts representing documents that will be accessible to the model if it is performing RAG249        (retrieval-augmented generation). If the template does not support RAG, this argument will have no250        effect. We recommend that each document should be a dict containing "title" and "text" keys. Please251        see the RAG section of the [chat templating guide](https://huggingface.co/docs/transformers/main/en/chat_templating#arguments-for-RAG)252        for examples of passing documents with chat templates.253    add_generation_prompt (bool, *optional*):254        If this is set, a prompt with the token(s) that indicate255        the start of an assistant message will be appended to the formatted output. This is useful when you want to generate a response from the model.256        Note that this argument will be passed to the chat template, and so it must be supported in the257        template for this argument to have any effect.258    continue_final_message (bool, *optional*):259        If this is set, the chat will be formatted so that the final260        message in the chat is open-ended, without any EOS tokens. The model will continue this message261        rather than starting a new one. This allows you to "prefill" part of262        the model's response for it. Cannot be used at the same time as `add_generation_prompt`.263    return_assistant_tokens_mask (`bool`, defaults to `False`):264        Whether to return a mask of the assistant generated tokens. For tokens generated by the assistant,265        the mask will contain 1. For user and system tokens, the mask will contain 0.266        This functionality is only available for chat templates that support it via the `{% generation %}` keyword.267    N�tools�	documentsF�add_generation_prompt�continue_final_message�return_assistant_tokens_mask)r;r<r=r>rirr?rUr@rjrBrkrArlrmrDrDrDrErh�
s268




rhc@s2eZ
dZUd
ZdZeeed<dZee	ed<dS)�ChatTemplateLoadKwargsa�
269    Keyword arguments used to load multimodal data in processor chat templates.270 271    num_frames (`int`, *optional*):272        Number of frames to sample uniformly. If not passed, the whole video is loaded.273    load_audio_from_video (`bool`, *optional*):274            Whether to use the audio track of input video. If `True` the audio track will be loaded and passed to the275            processor. This flag has no effect if the model doesn't support audio modality.276    i�>r\F�load_audio_from_videoN)277r;r<r=r>r\rrCr@rorArDrDrDrErn�
s278
279
rnc@s2eZ
dZUd
ZdZeeed<dZeeed<dS)�ProcessorChatTemplateKwargsa:
280    Keyword arguments for processor's `apply_chat_template`.281 282    tokenize (`bool`, *optional*, defaults to `False`):283        Whether to tokenize the output or not.284    return_dict (`bool`, defaults to `False`):285        Whether to return a dictionary with named outputs. Has no effect if tokenize is `False`.286    F�tokenize�return_dictN)	r;r<r=r>rqrrAr@rrrDrDrDrErp�
s287
	
rpc@s&eZ
dZUeed
<eed<eed<dS)�AllKwargsForChatTemplateZprocessor_kwargs�mm_load_kwargs�template_kwargsN)r;r<r=rar@rnrprDrDrDrErs�
s288


rsc@sreZ
dZUd
ZdZeeeed<dZ	eeeed<dZ289eeeed<dZeeeed<dd�Zd	d290�Z
dS)�MultiModalDataa�
291    Dataclass that holds extra useful data for processing292    multimodal data. Processors currently cannot return keys,293    unless it is used in model's forward. Thus we have helper294    methods that calculate and return useful data from processing295    input multimodals (images/videos).296    Note that this dataclass is aimed to be used only in vLLM297    and we might change its API in the future.298    N�num_image_tokens�num_video_tokens�num_audio_tokens�num_image_patchescCst||
�ot
||
�du
S�
N)�hasattr�getattr��self�keyrDrDrE�__contains__�
s
zMultiModalData.__contains__cCs*t||
�r299t
||
�St|jj�d
|
���
�
)Nz has no attribute )r|r}�AttributeError�	__class__r;r~rDrDrE�__getitem__�
s300
301

zMultiModalData.__getitem__)r;r<r=r>rwrr?rCr@rxryrzr�r�rDrDrDrErv�
s302303


rvc@s�eZ
dZUd
ZddgZddgZgZeee	d<dZ304dZdZe
Zdd	�Z				dWd305eedeeeeeeeefdeed
eedee
f306dd�Zdd�ZdXdeeeffdd�ZdXdefdd�ZdXdeeejffdd�Z dd�Z!dYde"d e"fd!d"�Z#e$d#eeejfde%eeefeeefffd$d%��
Z&e$d&eeeffd'd(��
Z'	dZd)e
d*eedeeeffd+d,�Z(e$					-d[d.e)e*d#eeejfd/eeeejfd0e"d1e"d2eeee"fd3ede*fd4d5��
Z+e$d\d7d8�
�
Z,e$d9d:��
Z-e.d;d<��
Z/d=d>�Z0d?d@�Z1e2dAdB��
Z3e.dCdD��
Z4e5dEdFdGdH�e5dIdJdKdL�	dZdMeeeeefeeeeeffdeedee6defdNdO��
�
Z7dXdPdQ�
Z8deedRdSdTeefdUdV�Z9dS)]r.za307    This is a mixin used to provide saving/loading functionality for all processor classes.308    �feature_extractor�	tokenizer�
chat_template�audio_tokenizer�optional_call_argsNc
	Os(
|jD],}|�
|d�}t|||�
|d
kr/|du
r/|�||�}t�r't|t�s/td|�d��
�
q|D]}||jv
rAt	d|�d��
�
q2t309|
|j�D]\}}||vrXt	d|�d��
�
|||<qHt|�
t|j�
kr}tdt|j�
�dd	�|j�
�d310t|
�
�d��
�
|�
�D]\}}|�||�
t|||�
q�dS)Nr�zTried to use `zW` for audio tokenization. However, this class is not registered for audio tokenization.zUnexpected keyword argument �
.z!Got multiple values for argument zThis processor requires z arguments: �, z. Got z arguments instead.)�optional_attributes�pop�setattr�check_argument_for_proper_classr(�311isinstancer,�312ValueError�313attributes�	TypeError�zip�len�join�items)	r�args�kwargsZoptional_attributeZoptional_attribute_value�proper_classr��arg�attribute_namerDrDrE�__init__�
s8314



315
��316

�

317


��

�zProcessorMixin.__init__�images�text�videos�audior�c
Ks�|
d
ur|d
ur|d
ur|d
urtd|j
j���
�
|j|jf
dt|d�r(|jjn
ii
|�
�
}|df|
df|df|dfd	�}i}|jD]%}t	||d
�}	||\}318}|319d
u
rg|	d
u
rg|	|320f
i||�
�
}|�321|�

qBt|�
S)322a7323        Main method to prepare for model inputs. This method forwards the each modality argument to its own processor324        along with `kwargs`. Please refer to the docstring of the each processor attributes for more information.325 326        Args:327            images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`):328                The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch329                tensor. Both channels-first and channels-last formats are supported.330            text (`TextInput`, `PreTokenizedInput`, `list[TextInput]`, `list[PreTokenizedInput]`, *optional*):331                The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings332                (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set333                `is_split_into_words=True` (to lift the ambiguity with a batch of sequences).334            videos (`np.ndarray`, `torch.Tensor`, `List[np.ndarray]`, `List[torch.Tensor]`):335                The video or batch of videos to be prepared. Each video can be a 4D NumPy array or PyTorch336                tensor, or a nested list of 3D frames. Both channels-first and channels-last formats are supported.337            audio (`np.ndarray`, `torch.Tensor`, `list[np.ndarray]`, `list[torch.Tensor]`):338                The audio or batch of audio to be prepared. Each audio can be a NumPy array or PyTorch339                tensor.340            return_tensors (`str` or [`~utils.TensorType`], *optional*):341                If set, will return tensors of a particular framework. Acceptable values are:342 343                - `'tf'`: Return TensorFlow `tf.constant` objects.344                - `'pt'`: Return PyTorch `torch.Tensor` objects.345                - `'np'`: Return NumPy `np.ndarray` objects.346                - `'jax'`: Return JAX `jnp.ndarray` objects.347 348        Returns:349            [`BatchFeature`]: A [`BatchFeature`] object with processed inputs in a dict format.350        Nz/You need to provide at least one input to call �tokenizer_init_kwargsr�rcrdrerf)r��image_processorZvideo_processorr�)r�r�r;�
_merge_kwargs�valid_processor_kwargsr|r��init_kwargsr�r}�updater)
rr�r�r�r�r�Zattribute_to_kwargsZoutputsr��	attributeZ351input_dataZinput_kwargsZattribute_outputrDrDrE�__call__s. %

���


�352




353
�zProcessorMixin.__call__csxt�|
�d
��}t
�||�}t|t�rt�f
dd�|D�
�
}n��|�
}t||�s:tdt|�
j�d|
�d|�d��
�
|S)z�354        Checks the passed argument's class against the expected transformers class. In case of an unexpected355        mismatch between expected and actual class, an error is raise. Otherwise, the proper retrieved class356        is returned.357        �_classc
3s"�|]}
|
du
r��|
�
V
qdSr{�
�get_possibly_dynamic_module��.0�
n�
rrDrE�	<genexpr>a�� zAProcessorMixin.check_argument_for_proper_class.<locals>.<genexpr>zReceived a z for argument z, but a z was expected.)	r}�AUTO_TO_BASE_CLASS_MAPPING�getr��tupler�r��typer;)rZ
argument_nameZargument�358class_namer�rDr�rEr�Ws359

360361

�z.ProcessorMixin.check_argument_for_proper_classT�returncs�t�
�j�
}t��j�
}t|j�
��d
g
7��r!�f
dd��D�
�d|vr(|d=d|vr/|d=d|vr6|d=d|vr=|d=�
f
dd	��
��fd362d�|��D�
}�
|�
}�sgd|vrg�j	j363j�j	jd
�}||d<�j364j|d<|S)z�365        Serializes this instance to a Python dictionary.366 367        Returns:368            `dict[str, Any]`: Dictionary of all the attributes that make up this processor instance.369        �auto_mapc
sg|]370}
|
�jj
v
r|
�qSrD)r�r�)r��
xr�rDrE�371<listcomp>~�z*ProcessorMixin.to_dict.<locals>.<listcomp>r�Zqformer_tokenizerZprotein_tokenizerr�c
sF|��D]\}
}t
|tj�r|��||
<qt
|t�r �|�
||
<q|S)
z�372            Numpy arrays are not serialiazable but can be in pre-processing dicts.373            This function casts arrays to list, recusring through the nested configs as well.374            )r�r�r]r^�tolistrU)�375dictionaryr��value)
�cast_array_to_listrDrEr��s

376

�
z2ProcessorMixin.to_dict.<locals>.cast_array_to_listc
sNi|]#\}
}|
�vr%|jj
dkr�
rt|t�r�
s|
t|t�r#|��n
|�qS)
ZBeamSearchDecoderCTC)r�r;r�r �to_dict)r��
k�
v)�
attrs_to_save�legacy_serializationrDrE�377<dictcomp>�s
����z*ProcessorMixin.to_dict.<locals>.<dictcomp>r���audio_tokenizer_class�audio_tokenizer_name_or_path�processor_class)
�copy�deepcopy�__dict__�inspect�	signaturer�r?�378parametersr�r�r�r;�name_or_path)rr��output�sig�audio_tokenizer_dictrD)r�r�r�rrEr�ls4379380







�
�zProcessorMixin.to_dictcCs |j|
d
�
}t
j|ddd�dS)z�381        Serializes this instance to a JSON string.382 383        Returns:384            `str`: String containing all the attributes that make up this feature_extractor instance in JSON format.385        �
r��T��indent�	sort_keys�
386)r��json�dumps)rr�r�rDrDrE�to_json_string�szProcessorMixin.to_json_string�json_file_pathcCsFt|
d
dd��}|�
|j|d�
�

Wd�
dS1sw



Y
dS)z�387        Save this instance to a JSON file.388 389        Args:390            json_file_path (`str` or `os.PathLike`):391                Path to the JSON file in which this processor instance's parameters will be saved.392        �
w�utf-8�
�encodingr�N)�open�writer�)rr�r��writerrDrDrE�to_json_file�s
"�zProcessorMixin.to_json_filec
s:�f
d
d��jD�
}
d�
|
�
}
�jj�d|
�d�����S)Nc
	s&g|]}
d|
�d
tt
�|
��
���qS)z- z: )�reprr})r��namer�rDrEr��s&z+ProcessorMixin.__repr__.<locals>.<listcomp>r�z:393z394 395)r�r�r�r;r�)rZattributes_reprrDr�rE�__repr__�s
396

zProcessorMixin.__repr__F�push_to_hubr�c 	s�|�d
d�}|du
rt
�dt�
|�d�
du
rtd�
�
||d<tj|
dd�
|rI|�dd�}|�d	|
�tj	j397�
d398�}�j|f
i|�
�
}��|
�
}�j
du
rk�f
dd��jD�
}	d
d�|	D�
}399|400���

t�|
|401d�
|�dd�}�jD]7}|dkr�t�|�}
t|
d�r�|
��jj�

|
j|
|d�
qt|r�t�|�}
t|
d�r�|
��jj�

|
�|
�

qt�j
du
rňjD]}t�|�}
t|
t�r�|
jd=q�tj	�|
t�}tj	�|
t�}tj	�|
t�}tj	�|
t�}�jdu
�
r�|�dd�}t�jt �}|�
r |�
r t!|ddd��}|�"�j�

Wd�
n	1�
sw



Y
t#�$d|���

n�|�
r�|�
s��j�%�D]j\}}|dk�
r^t!|ddd��}|�"�jd�

Wd�
n	1�
sOw



Y
t#�$d|���

�
q+tj|dd�
tj	�||�d��}t!|ddd��
}|�"|�

Wd�
n	1�
s�w



Y
t#�$d|���

�
q+n@|�
r�t&j'd�ji
ddd�d}t!|ddd��
}|�"|�

Wd�
n	1�
s�w



Y
t#�$d|���

n402�jdu
�
r�td�
�
|�rHtj	�|
t(�}��)�}t*|�+��
dh
k�
r���,|�

t#�$d |���

t*|�+��
dh
k�r	g}n|g
}�j-du
�rG�j-jj}�j-j.}||d!�}t&j'|ddd�d}t!|ddd��
}|�"|�

Wd�
n	1�sBw



Y
n�j,|d"d#�
t#�$d |���

|g
}|�rj�j/|
||||�d�
d$�
|S)%aU403        Saves the attributes of this processor (feature extractor, tokenizer...) in the specified directory so that it404        can be reloaded using the [`~ProcessorMixin.from_pretrained`] method.405 406        <Tip>407 408        This class method is simply calling [`~feature_extraction_utils.FeatureExtractionMixin.save_pretrained`] and409        [`~tokenization_utils_base.PreTrainedTokenizerBase.save_pretrained`]. Please refer to the docstrings of the410        methods above for more information.411 412        </Tip>413 414        Args:415            save_directory (`str` or `os.PathLike`):416                Directory where the feature extractor JSON file and the tokenizer files will be saved (directory will417                be created if it does not exist).418            push_to_hub (`bool`, *optional*, defaults to `False`):419                Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the420                repository you want to push to with `repo_id` (will default to the name of `save_directory` in your421                namespace).422            legacy_serialization (`bool`, *optional*, defaults to `True`):423                Whether or not to save processor attributes in separate config files (legacy) or in processor's config424                file as a nested dict. Saving all attributes in a single dict will become the default in future versions.425                Set to `legacy_serialization=True` until then.426            kwargs (`dict[str, Any]`, *optional*):427                Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method.428        �use_auth_tokenN�rThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.�token�V`token` and `use_auth_token` are both specified. Please set only the argument `token`.T)
�exist_ok�commit_message�repo_id�����c
sg|]}
t�|
��qSrD)
r})r�r�r�rDrEr���z2ProcessorMixin.save_pretrained.<locals>.<listcomp>c
Ss g|]}
t|
t
�r|
jn
|
�qSrD)r�rr�)r��
arDrDrEr��s )
�config�save_jinja_filesr��_set_processor_class)
r�r�r�r�r�zchat template saved in �default�.jinjar�r�r�r�z�Multiple chat templates are not supported in the legacy format. Please save them as separate files using the `save_jinja_files` argument.r�zprocessor saved in r�Fr�)r�r�)0r��warnings�warn�
FutureWarningr�r��os�makedirs�split�path�sepZ_create_repoZ_get_files_timestamps�_auto_classr��appendr
r}r|r�r�r;�save_pretrainedr�rr�r�rrrrr�rBr�r��logger�infor�r�r�rr��set�keysr�r�r�Z_upload_modified_files) rZsave_directoryr�r�r�r�r�r�Zfiles_timestamps�attrsZconfigsr�r�r�Zoutput_processor_fileZoutput_chat_template_file_jinjaZ output_chat_template_file_legacyZchat_template_dirZis_single_template�
f�
template_name�templateZtemplate_filepathZchat_template_json_stringr�Zoutput_audio_tokenizer_file�processor_dictZreturn_filesr�r�r�Zaudio_tokenizer_jsonrDr�rEr
�s�


�

�


429
430

431

432433
434


435
436
437
�438439440
441

�


�



�
442


�


���
�

�

443


444


�

��







�zProcessorMixin.save_pretrained�pretrained_model_name_or_pathcst�
|�
}|�d
d��|�dd��
|�dd��|�dd��|�dd��|�dd��|�d	d��|�d445d��|�dd�}|�d
d�}d|d��	|du
rN|�	d<t�rZ�sZt�d�

d�t��
�tj�	��
}tj�	��
rqtj�446�t�}i}i}	tj���
r��}447d}d}d}
d}n�t
��
r��}t��
}448d}d}d}
n�|r�t�t�}|��r�|�d�
D]}|j}t�d|j��||<q�n"zt�����d�D]}t�d|�d�||<q�Wn	ty�


Yn
wt}zVt�|��
�����	��dd�}449t�t��
�����	��dd�}t�t��
�����	��dd�}��
��������	f450dd�|��D�
}	t�t��
�����	��dd�}
Wnt�
y;


�t�
yP


td��d��dt�d��
�
w|du
�
r�t|dd��}t� |�!��
}d |d!i
}|	�
rqt"d"�
�
Wd�
n	1�
s|w



Y
n,d#d�|	��D�
}|du
�
r�t|d$dd��}|�!�|d <Wd�
n	1�
s�w



Y
t#|t$��
r�d |v�
r�t%|�
d%k�
r�|d }|�
r�||d!<|451du�
r�i}n4z"t|452dd��}|�!�}Wd�
n	1�
s�w



Y
t� |�
}Wntj&�y


td&|453�d'��
�
w|�rt�d(|454���

nt�d(|�d)|455���

d!|v�r/|d!du
�r/t�'d*�

d!|v�r;|�d!�
|d!<|
du
�sEd+|v�ru|
du
�r[t|
d$dd�}|�!�}t� |�
}n|d+}|�(|d,�
}|d-}|j)|f
i|�
�
|d+<|j*D]	}|�|d�
�qx||fS).a456        From a `pretrained_model_name_or_path`, resolve to a dictionary of parameters, to be used for instantiating a457        processor of type [`~processing_utils.ProcessingMixin`] using `from_args_and_dict`.458 459        Parameters:460            pretrained_model_name_or_path (`str` or `os.PathLike`):461                The identifier of the pre-trained checkpoint from which we want the dictionary of parameters.462            subfolder (`str`, *optional*, defaults to `""`):463                In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can464                specify the folder name here.465 466        Returns:467            `tuple[Dict, Dict]`: The dictionary(ies) that will be used to instantiate the processor object.468        �	cache_dirN�force_downloadF�resume_download�proxiesr��local_files_only�revision�	subfolder�Z_from_pipelineZ469_from_auto�	processor)�	file_type�from_auto_classZusing_pipelinez+Offline mode: forcing local_files_only=TrueTz*.jinja�
/)r
r
r
r�r��470r
r
r
r
r
r��471user_agentr
r
Z%_raise_exceptions_for_missing_entriesc
s2i|]\}
}|
t�|��
�����	��dd
��qS)Fr
)
r"�r�r
�
template_file�472r
r
r
r
r
r
r
r
r�r
rDrEr��s"�











��z5ProcessorMixin.get_processor_dict.<locals>.<dictcomp>zCan't load processor for 'z�'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z2' is the correct path to a directory containing a z filer�r�r�r�a
Cannot load chat template due to conflicting files - this checkpoint combines a legacy chat_template.json file with separate template files, which is not supported. To resolve this error, replace the legacy chat_template.json file with a modern chat_template.jinja file.c
Ss$i|]\}
}|
t|dd
d��
��qS)�
rr�r�)r��readr
rDrDrEr�9s��r 
r473z"It looks like the config file at 'z' is not a valid JSON file.zloading configuration file z from cache at z�Chat templates should be in a 'chat_template.jinja' file but found key='chat_template' in the processor's config. Make sure to move your template to its own file.r�r�r�)+r�r�r�r&r
r
rBr�r
�isdirr�r�isfiler'r%rr�is_dir�glob�stemr�r)r	r"rrr�r�OSError�	Exceptionr�r��loadsr!
r�r�rUr��JSONDecodeError�warning_oncer��from_pretrainedr�)�clsr
r�Zaudio_tokenizer_kwargsZ
from_pipeliner
�is_localZprocessor_fileZadditional_chat_template_filesZ'resolved_additional_chat_template_filesZresolved_processor_fileZresolved_chat_template_fileZresolved_raw_chat_template_fileZresolved_audio_tokenizer_fileZtemplate_dirr
r
r
�readerZchat_template_jsonZchat_templatesr

r�r�r�Zaudio_tokenizer_pathr�rDr
rE�get_processor_dictzst
474







475

476477















478




�





479��
�











�











�











��











480�

����481 482




�����483

�$

484485
�
486
��


�487
488






�
489�490
z!ProcessorMixin.get_processor_dictr

cs�|��}|�
d
d�}d|vr|d=d|vr|d=|�|�

|jjjd|jjj�dd�}|j||d�\}���fdd	�t|�
D�
�
�
f
d491d�t��
D�
�|�i��
�
}t	�492d|���

|rb||fS|S)
a�493        Instantiates a type of [`~processing_utils.ProcessingMixin`] from a Python dictionary of parameters.494 495        Args:496            processor_dict (`dict[str, Any]`):497                Dictionary that will be used to instantiate the processor object. Such a dictionary can be498                retrieved from a pretrained checkpoint by leveraging the499                [`~processing_utils.ProcessingMixin.to_dict`] method.500            kwargs (`dict[str, Any]`):501                Additional parameters from which to initialize the processor object.502 503        Returns:504            [`~processing_utils.ProcessingMixin`]: The processor object instantiated from those505            parameters.506        �return_unused_kwargsFr�r�Nr507)�processor_config�valid_kwargsc
s0i|]\}
}|�
vr|
t��
kr|
�
�
|�
�qSrD)r�r��r��
ir�)r�r3
rDrEr��s508
509�z5ProcessorMixin.from_args_and_dict.<locals>.<dictcomp>c
sg|]510\}
}��|
|��qSrD)
r�r4
)
�args_to_updaterDrEr��r�z5ProcessorMixin.from_args_and_dict.<locals>.<listcomp>z511Processor )r�r�r�r��__code__�co_varnames�co_argcount�validate_init_kwargs�	enumerater
r
)r-
r�r

r�r1
Zaccepted_args_and_kwargs�
unused_kwargsr
rD)r�r6
r3
rE�from_args_and_dict}s(


512 
513��

z!ProcessorMixin.from_args_and_dict�ModelProcessorKwargsr�cs2iiiiid
�}iiiiid
��hd�
}t�}�D]4}|
j
�|i����|<|
j|jD] }|du
rL||vrLt|j|�rBt|j|�n||}	|	�||<q,q|���

t|�
t|�
}514|�	�D]L\}}|
j|jD]A}||vr�||�515|d�}|dkr�||516vr�td|�d|�d��
�
n
||vr�|�|d�}nd}t|t
�r�|dkr�|||<|�|�

qiq_t�f
dd	�|D�
�
r�|�	�D] \}}
|�vr�|
�	�D]\}}||v
r�||||<|�|�

q�q�n+|�	�D]&\}}||v
�
r||
jd517jvr�||d518|<q�||v
�
rt�d|�d��

q�|��D]519}|�|d520�

�
q|S)
a521        Method to merge dictionaries of kwargs cleanly separated by modality within a Processor instance.522        The order of operations is as follows:523            1) kwargs passed as before have highest priority to preserve BC.524                ```python525                high_priority_kwargs = {"crop_size" = {"height": 222, "width": 222}, "padding" = "max_length"}526                processor(..., **high_priority_kwargs)527                ```528            2) kwargs passed as modality-specific kwargs have second priority. This is the recommended API.529                ```python530                processor(..., text_kwargs={"padding": "max_length"}, images_kwargs={"crop_size": {"height": 222, "width": 222}}})531                ```532            3) kwargs passed during instantiation of a modality processor have fourth priority.533                ```python534                tokenizer = tokenizer_class(..., {"padding": "max_length"})535                image_processor = image_processor_class(...)536                processor(tokenizer, image_processor) # will pass max_length unless overridden by kwargs at call537                ```538            4) defaults kwargs specified at processor level have lowest priority.539                ```python540                class MyProcessingKwargs(ProcessingKwargs, CommonKwargs, TextKwargs, ImagesKwargs, total=False):541                    _defaults = {542                        "text_kwargs": {543                            "padding": "max_length",544                            "max_length": 64,545                        },546                    }547                ```548        Args:549            ModelProcessorKwargs (`ProcessingKwargs`):550                Typed dictionary of kwargs specifically required by the model passed.551            tokenizer_init_kwargs (`Dict`, *optional*):552                Dictionary of kwargs the tokenizer was instantiated with and need to take precedence over defaults.553 554        Returns:555            output_kwargs (`Dict`):556                Dictionary of per-modality kwargs to be passed to each modality-specific processor.557 558        )rcrdrfrerb>r�r�r�r�NZ	__empty__zKeyword argument z+ was passed two times:559in a dictionary for z and as a **kwarg.c
3s�|]}
|
�vV
qdSr{rD)r�r��
Zdefault_kwargsrDrEr�s�z/ProcessorMixin._merge_kwargs.<locals>.<genexpr>rbzKeyword argument `zA` is not a valid argument for this processor and will be ignored.)r
rgr�r�r@r|r�r}r�r�r�r�r�rB�add�anyr
r+
�values)rr>
r�r�Z
output_kwargsZpossible_modality_keywordsZ	used_keys�modalityZmodality_keyr�Znon_modality_kwargsZoutput_kwargZkwarg_valueZsubdict�subkeyZsubvaluer��kwargrDr?
rEr��s�/



�	



�

560����561





���

562
��



563
��564


565

566
��

zProcessorMixin._merge_kwargs�mainr-
r
r
r
r�r
cKs�||d
<||d<||d<||d<|�dd�}|du
r*t
�dt�
|du
r(td�
�
|}|du
r2||d	<|j|
f
i|�
�
}	|j|
f
i|�
�
\}567}|j|	|568fi|�
�
S)569a[570        Instantiate a processor associated with a pretrained model.571 572        <Tip>573 574        This class method is simply calling the feature extractor575        [`~feature_extraction_utils.FeatureExtractionMixin.from_pretrained`], image processor576        [`~image_processing_utils.ImageProcessingMixin`] and the tokenizer577        [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`] methods. Please refer to the docstrings of the578        methods above for more information.579 580        </Tip>581 582        Args:583            pretrained_model_name_or_path (`str` or `os.PathLike`):584                This can be either:585 586                - a string, the *model id* of a pretrained feature_extractor hosted inside a model repo on587                  huggingface.co.588                - a path to a *directory* containing a feature extractor file saved using the589                  [`~SequenceFeatureExtractor.save_pretrained`] method, e.g., `./my_model_directory/`.590                - a path or url to a saved feature extractor JSON *file*, e.g.,591                  `./my_model_directory/preprocessor_config.json`.592            **kwargs593                Additional keyword arguments passed along to both594                [`~feature_extraction_utils.FeatureExtractionMixin.from_pretrained`] and595                [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`].596        r
r
r
r
r�Nr�r�r�)r�r�r�r�r��_get_arguments_from_pretrainedr0
r=
)r-
r
r
r
r
r�r
r�r�r�r

rDrDrEr,
7s('






�

�


zProcessorMixin.from_pretrained�
AutoProcessorcCsDt|
t
�s|
j}
d
dlm
m}
t||
�st|
�d��
�
|
|_dS)as
597        Register this class with a given auto class. This should only be used for custom feature extractors as the ones598        in the library are already mapped with `AutoProcessor`.599 600 601 602        Args:603            auto_class (`str` or `type`, *optional*, defaults to `"AutoProcessor"`):604                The auto class to register this new feature extractor with.605        r
Nz is not a valid auto class.)	r�rBr;Ztransformers.models.auto�models�autor|r�r
)r-
Z606auto_classZauto_modulerDrDrE�register_for_auto_classvs607
608
609z&ProcessorMixin.register_for_auto_classc	s�g}�jD]V}t
�|�d
��}t|t�rJt�f
dd�|D�
�
}|dkr2|�d�
}|dur1t�d�

n|�dd�}|rE|d	du
rE|d	}n610|d611}n��|�
}|�|j	|
f
i|�
�
�

q|S)a�612        Identify and instantiate the subcomponents of Processor classes, like image processors and613        tokenizers. This method uses the Processor attributes like `tokenizer_class` to figure out what class those614        subcomponents should be. Note that any subcomponents must either be library classes that are accessible in615        the `transformers` root, or they must be custom code that has been registered with the relevant autoclass,616        via methods like `AutoTokenizer.register()`. If neither of these conditions are fulfilled, this method617        will be unable to find the relevant subcomponent class and will raise an error.618        r�c
3s&�|]}
|
du
r
��|
�
n
dV
qdSr{r�r��
r-
rDrEr��s�$z@ProcessorMixin._get_arguments_from_pretrained.<locals>.<genexpr>r��use_fastNaC
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.Tr619r
)620r�r}r�r�r�r
r+
r�r
r,
)	r-
r
r�r�r�r��classesrM
Zattribute_classrDrL
rErG
�s&621 622

623


624


��
625
626627z-ProcessorMixin._get_arguments_from_pretrainedc
Cs�tt
|�r628tt
|�St
jt
jt
jt
jt
jg}
|
D]5}|j�	�D]-}t629|t�r=|D]}|du
r;|j|kr;|


Sq(q|du
rL|j|krL|

Sqqt
d
|�d��
�
)NzCould not find module z� in `transformers`. If this is a custom class, it should be registered using the relevant `AutoClass.register()` function so that other functions can find it!)r|�transformers_moduler}ZIMAGE_PROCESSOR_MAPPINGZVIDEO_PROCESSOR_MAPPINGZTOKENIZER_MAPPINGZFEATURE_EXTRACTOR_MAPPINGZ$MODEL_FOR_AUDIO_TOKENIZATION_MAPPINGZ_extra_contentrB
r�r�r;r�)�module_nameZlookup_locationsZlookup_locationZcustom_classZcustom_subclassrDrDrEr��s.630631




�
632



��
��633
�z*ProcessorMixin.get_possibly_dynamic_modulec
O�0t|d
�st
d|jj�d��
�
|jj|
i|�
�
S)z�634        This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please635        refer to the docstring of this method for more information.636        r�zCannot batch decode text: � has no tokenizer.)r|r�r�r;r��batch_decode�rr�r�rDrDrErS
��637

zProcessorMixin.batch_decodec
OrQ
)z�638        This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to639        the docstring of this method for more information.640        r�zCannot decode text: rR
)r|r�r�r;r��decoderT
rDrDrErV
�rU
zProcessorMixin.decodec
Cs4g}
|jD]}t
||d�}t
|d
�}|
�|�

q|
S)N�model_input_names)r�r}�extend)rrW
r�r�Zattr_input_namesrDrDrErW
�s641

642


z ProcessorMixin.model_input_namescs`t��
��
}t|
�
}||}||@}|r�f
d
d�|D�
n
i}|r*�f
dd�|D�
n
i}
||
fS)Nc
�i|]}
|
�|
�qSrDrD�r�r��
r2
rDrEr��r�z7ProcessorMixin.validate_init_kwargs.<locals>.<dictcomp>c
rY
rDrDrZ
r[
rDrEr��r�)r
r
)r2
r3
Zkwargs_from_configZvalid_kwargs_setZunused_keysZ643valid_keysr<
rDr[
rEr:
�s


z#ProcessorMixin.validate_init_kwargsZ	video_fpsz4.58rY)�version�new_name�video_load_backendz4.59zd. This function will use `torchcodec` by default, or `torchvision` if `torchcodec` is not installed.)r\
Zadditional_message�conversationc-	Ks�|d
ur5t|j
t�rd|j
vr|j
d}n2t|j
t�r(tdd�|j
���
���
�
|j
d
u
r1|j
}ntd�
�
t|j
t�rF||j
vrF|j
|}n
	t|d�oS|jjj	�644d�
}|�dd	�rn|�d645d	�rdtd�
�
|�dd	�rntd
�
�
|�dd	�r~|sztd�
�
d|d<iid�}|D]+}tj
|j
D]"}tj
|}t||d
�}	|�||	�}646|647d
u
r�t|648t�s�|649|||<q�q�|�dd
�
|d�|�

t|
ttf�r�t|
dttf�s�t|
dd�r�d}|
}nd	}|
g
}|d�dd	�}
|d�dd	�}|d}|
�
rjgg}}g}|D]j}
gg}}|
D]V}dd�|dD�
}dd�|dD�
}dd�|D�
}|�|�

dd�|D�
}|�|�

|d�
sL|D]}|�t||dd ��

�
q;�
q|D]}|�t||dd ��

�
qN�
q|�|�

|�|�

q�td1||d!�|d�
|jj�
�
\}}|�
s�|d}|
�r}|�
r�|dn
|}|jjd
u
�
r�|�|jj�
�
r�d	|d"<d#|v
�
r�|�d$�
d
u
�
s�|�d%�
d
u
�
r�d|d#<td&d'�|D�
�
}td(d'�|D�
�
}|d1||�
r�|n
d
|�
r�|n
d
|�
r�|n
d
d)�|�
�
} |�ry|d�dd	��rwg}!| �d*�
}"| d+}#tt|#�
�
D]d}$dg
t|#|$�
}%|"|$}&d,d�|&D�
}'||$D]C\}(})t�|'|(�}*t�|'|)�}+|*dk�rI|&|*d|(k
�rH|&|*d-k�sKn

�qt|*|+�rR|+nt|#|$�
�D]},d-|%|,<�qY�q|!�|%�

�q|!| d.<| j|�d/�
d0�

| S| d+S|S)2a�650        Similar to the `apply_chat_template` method on tokenizers, this method applies a Jinja template to input651        conversations to turn them into a single tokenizable string.652 653        The input is expected to be in the following format, where each message content is a list consisting of text and654        optionally image or video inputs. One can also provide an image, video, URL or local path which will be used to form655        `pixel_values` when `return_dict=True`. If not provided, one will get only the formatted text, optionally tokenized text.656 657        conversation = [658            {659                "role": "user",660                "content": [661                    {"type": "image", "url": "https://www.ilankelman.org/stopsigns/australia.jpg"},662                    {"type": "text", "text": "Please describe this image in detail."},663                ],664            },665        ]666 667        Args:668            conversation (`Union[list[Dict, [str, str]], list[list[dict[str, str]]]]`):669                The conversation to format.670            chat_template (`Optional[str]`, *optional*):671                The Jinja template to use for formatting the conversation. If not provided, the tokenizer's672                chat template is used.673        Nr�z�The processor has multiple chat templates but none of them are named "default". You need to specify which one to use by passing the `chat_template` argument. Available templates are: r�zTCannot use apply_chat_template because this processor does not have a chat template.r�ZFastrlFrka
continue_final_message and add_generation_prompt are not compatible. Use continue_final_message when you want the model to continue the final message, and add_generation_prompt when you want to add a header that will prompt it to start a new assistant message instead.rmzKcontinue_final_message is not compatible with return_assistant_tokens_mask.z�`return_assistant_tokens_mask` is not possible with slow tokenizers. Make sure you have `tokenizers` installed. If the error persists, open an issue to support a Fast tokenizer for your model.Tr9)rtrur^
rur
�contentrqrrrtc
Ssg|]674}
|
dd
vr|
�qS)r�)�image�videorD)r�r`
rDrDrEr�hr�z6ProcessorMixin.apply_chat_template.<locals>.<listcomp>c
S�2g|]}
dD]}||
vr|
d
dkr|
|�qqS))r��urlr
r�r�rD)r�r`
r�rDrDrEr�i�
�
�c
Src
))ra
rd
r
�base64r�ra
rD�r�Zvision_infor�rDrDrEr�ore
c
Src
))rb
rd
r
r�rb
rDrg
rDrDrEr�vre
ror\)
r\)�
conversationsr�r3rXrYrZc
s�"�|]}
|
D]}|du
V
qqdSr{rD)r�Zim_listZimrDrDrEr��r�z5ProcessorMixin.apply_chat_template.<locals>.<genexpr>c
sri
r{rD)r�Zvid_listZvidrDrDrEr��r�)r�r�r�r��offset_mapping�	input_idsc
Ssg|]\}
}|
�qSrDrD)r��start�endrDrDrEr��sr675�assistant_masksr`)
Ztensor_typerD) r�r�rUr�r�r
r|r�r�r;�endswithr�rsr@r}r�r�r?r�rX
r
rrZspecial_tokens_mapZ	bos_token�676startswithrA
�ranger��bisect�bisect_leftZconvert_to_tensors)-rr_
r�r�Zis_tokenizers_fastZprocessed_kwargsZ677kwarg_typer�Zkwarg_type_defaults�
default_valuer�Z678is_batchedrh
rqrrrtZbatch_imagesZbatch_videosZbatch_audiosr�r��messageZvisualsZaudio_fnamesZimage_fnamesZvideo_fnames�fname�promptZgeneration_indicesZ
single_promptZimages_existZvideos_exist�outrn
rj
rk
r5
Zcurrent_mask�offsetsZ
offset_startsZassistant_start_charZassistant_end_char�	start_pos�end_pos�token_idrDrDrE�apply_chat_template�s
%




��679

�


�



�
�
680




��
�
�



681


682



��683
�684685

�
�686


��687�


688 






��

689








690*
 

�



z"ProcessorMixin.apply_chat_templatecKs|jj
|
f
d
|i
|�
�
S)a�691        Post-process the output of a vlm to decode the text.692 693        Args:694            generated_outputs (`torch.Tensor` or `np.ndarray`):695                The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)`696                or `(sequence_length,)`.697            skip_special_tokens (`bool`, *optional*, defaults to `True`):698                Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method.699            **kwargs:700                Additional arguments to be passed to the tokenizer's `batch_decode method`.701 702        Returns:703            `list[str]`: The decoded text.704        �skip_special_tokens)r�rS
)rZgenerated_outputsr~
r�rDrDrE�post_process_image_text_to_text�sz.ProcessorMixin.post_process_image_text_to_text�text_inputsr�705modalitiesc	sz|D]8}t||�d
���
t||�d����f
dd�|dD�
}�
f
dd�|
D�
}||kr:t
d|�d|�d	|�d706��
�
qdS)z�707        Checks that number of special tokens in text and processed text is same. The count can be different708        if tokenized text was truncated, leading to issues in model code.709        �_tokenZ	_token_idc
sg|]	}
t|
�
�
��
�qSrD)r?�count)r�Zids)
r|
rDrEr��sz;ProcessorMixin._check_special_mm_tokens.<locals>.<listcomp>rk
c
sg|]}
|
���
�qSrD)
r�
)r��sample)
�	token_strrDrEr��r�z
Mismatch in `z4` token count between text and `input_ids`. Got ids=z710 and text=z^. Likely due to `truncation='max_length'`. Please disable truncation or increase `max_length`.N)r}r�)rr�r�
r�
rC
Z	ids_countZ711text_countrD)r|
r�
rE�_check_special_mm_tokens�s





���z'ProcessorMixin._check_special_mm_tokens)NNNN)
T)FTr{)NFFNrF
)
rH
):r;r<r=r>r�r�r�r?rBr@Zfeature_extractor_classZtokenizer_classr
rar�r�rrrrrrr�Unpackr�r�rUrr�r�r��PathLiker�r�rAr
�classmethodr�r0
r=
r�r�r-r,
rK
rG
�staticmethodr�rS
rV
�propertyrW
r:
r+rsr}
r
r�
rDrDrDrEr.�
s�712





(


�����713�>D0

��
:���714715�



�
�������	�>
716
$717
		718
719




��&����720W"r
rH
zprocessor files)�objectZobject_classZobject_files)[r>rr
r�r�r�r��sys�typingr��dataclassesr�pathlibrrrrrr�numpyr]Ztyping_extensionsZhuggingface_hub.errorsr	Zaudio_utilsrrZdynamic_module_utilsr
Zfeature_extraction_utilsrZimage_utilsrrrZutils.chat_template_utilsrZvideo_utilsrrrZtokenization_utils_baserrrrr�utilsrrrrrr r!r"r#r$r%r&r'r(r)r*Zutils.deprecationr+Zmodeling_utilsr,Z721get_loggerr;r
r-�__file__�parentrO
r��version_infor�
r2rGrWr[r_rarhrnrprsrvr.r��formatrDrDrDrE�<module>
s�


















H
722


�723
=7C':'



��