Apppublic
XaviXva/Video-LLaVA

sourceHugging Faceapache-2.0updated 3y agoView on Hugging Face
0likes
rootllavamodelmultimodal_encoderlanguagebinddepth__pycache__modeling_depth.cpython-310.pyc
modeling_depth.cpython-310.pyc442 linesDownload Raw Back to __pycache__
1o

2 eG��	@s�ddlZddlmZmZmZddlZddlmZddlm	Z	m3Z4ddlmZddlm
ZddlmZmZddlmZmZdd	lmZmZmZmZmZmZmZmZmZdd5lm Z m!Z!ddl"m#Z#m$Z$m%Z%Gd
d�dej&�Z'Gdd�dej&�Z(Gdd�de�Z)dZ*dZ+dZ,dZ-Gdd�dej&�Z.	d+dej/dej0dej1de2fdd�Z3Gdd �d ej&�Z4ed!e*�Gd"d#�d#e)��Z5Gd$d%�d%ej&�Z6ed&e*�Gd'd(�d(e)��Z7ee*�Gd)d*�d*e)��Z8dS),�N)�Optional�Tuple�Union)�	rearrange)�6LoraConfig�get_peft_model)�nn)�7functional)�PreTrainedModel�add_start_docstrings)�BaseModelOutput�BaseModelOutputWithPooling)	�CLIPMLP�
CLIPAttention�CLIPTextEmbeddings�CLIPVisionEmbeddings�CLIPVisionModelWithProjection�CLIPTextModelWithProjection�_expand_mask�8CLIPOutput�	clip_loss)�%add_start_docstrings_to_model_forward�replace_return_docstrings�)�LanguageBindDepthConfig�CLIPVisionConfig�CLIPTextConfigcs*eZdZdZd�fdd�	Zdd�Z�ZS)�PatchDropoutz*9    https://arxiv.org/abs/2212.0079410    Tcs6t���d|krdksJ�J�||_||_dS)Nr��?)�super�__init__�prob�exclude_first_token)�selfr!r"��	__class__��LD:\LLaVA\llava\model\multimodal_encoder\languagebind\depth\modeling_depth.pyr s1112zPatchDropout.__init__cCs4|jr|jdkr13|S|jr#|dd�dd�f|dd�dd�f}}ntj�tj|dd�dd�f�}|��d}|��d}t�|�}|d}d|j}t	dt14||��}	|dkrit�||�}15|16j|	dd�j
}nt�||�}17|18j|	dd�j
}|�d��d|d�}t|d�}|||f}|jr�tj||fdd�}|S)N�rr).N�������dimzb t n -> (b t) n)�trainingr!r"�torch�jit�annotate�Tensor�size�arange�max�int�randn�topk�indices�	unsqueeze�repeatr�cat)r#�x�B�T�19cls_tokens�batch�20num_tokens�
batch_indices�	keep_prob�num_patches_keep�rand�patch_indices_keepr&r&r'�forwards,, 212223zPatchDropout.forward)T)�__name__�24__module__�__qualname__�__doc__r rF�
__classcell__r&r&r$r'rsrcsTeZdZdef�fdd�Z	ddejdejdejdeed	e	ej25f26d27d�Z�ZS)
�CLIPEncoderLayer�configcs�t���|j|_t|�|_tj|j|jd�|_	t28|�|_tj|j|jd�|_|j
|_
|j
ro|j|_t�t�d|j|j��|_tjj|j|jdd�|j|_t|�|_tj|j|jd�|_t29|�|_tj|j|jd�|_dSdS)N��epsr�࿩�std)rr �hidden_size�	embed_dimr�	self_attnr�	LayerNorm�layer_norm_eps�layer_norm1r�mlp�layer_norm2�
add_time_attn�30num_frames�t�	Parameterr-�zeros�temporal_embedding�init�normal_�
temporal_attn�temporal_layer_norm1�temporal_mlp�temporal_layer_norm2�r#rMr$r&r'r Bs"3132333435�zCLIPEncoderLayer.__init__F�
hidden_states�attention_mask�causal_attention_mask�output_attentions�returncCsF|jrq|j\}}}|j}|dkr3|jd}t|d|d�}||jdd�d|�dd�f}t|d|d�}|}	t|d|d�}|�|�}|j||||d�\}}36|	t|d|d�}|}	t|d|d�}|�|�}|�|�}|	t|d|d�}|}	|�	|�}|j37||||d�\}}38|	|}|}	|�|�}|�|�}|	|}|f}|r�||39f7}|S)aI40        Args:41            hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)`42            attention_mask (`torch.FloatTensor`): attention mask of size43                `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.44                `(config.encoder_attention_heads,)`.45            output_attentions (`bool`, *optional*):46                Whether or not to return the attentions tensors of all attention layers. See `attentions` under47                returned tensors for more detail.48        rz(b t) n d -> (b n) t d)r]Nz(b n) t d -> (b t) n d)�n)rhrirjrk)
r[�shaper]rr`rdrcrfrerXrUrZrY)r#rhrirjrk�btrm�dr]�residual�attn_weights�outputsr&r&r'rFVsP49 5051�52535455�565758zCLIPEncoderLayer.forward�F)
rGrHrIrr r-r0r�boolr�FloatTensorrFrKr&r&r$r'rLAs������rLc@s4eZdZdZeZdZdZdgZdd�Z	ddd	�Z59d60S)�CLIPPreTrainedModelz�61    An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained62    models.63    �clipT�position_idscCs�|jj}t|t�r#|jjjjd|dd�|jjjjd|dd��nt|t	�rX|jj}t64jj|jd|j
d|d�t65jj|jj|jj|d�t66jj|jj|jj|d�n�t|t�r�|jj}|j
dd|jjd|}|j
d|}t67jj|jj|d�t68jj|jj|d�t69jj|jj|d�t70jj|jj|d�n�t|t�r�|jj}|jjdd|jjd|}d|jjd|}t71jj|jj|d�t72jj|jj|d�nZt|t��rt73jj|jj|jd|jjd�t74jj|jj|jd|jjd�n1t|t��rt75jj|jj|jjd|jjd�nt|t ��r2t76jj|jj|jjd|jjd�t|t77j!��rF|j"j�#�|jj�$d�t|t78j%��r[|j"du�r]|j"j�#�dSdSdS)	zInitialize the weightsr(g{�G�z�?)�meanrRrPrQ�rN)&rM�initializer_factor�79isinstancer�token_embedding�weight�datarb�position_embeddingrrra�class_embeddingrT�patch_embedding�initializer_ranger�num_hidden_layers�q_proj�k_proj�v_proj�out_projrrS�fc1�fc2�LanguageBindDepth�text_projection�text_embed_dim�visual_projection�vision_embed_dimrrrV�bias�zero_�fill_�Linear)r#�module�factor�in_proj_std�out_proj_std�fc_stdr&r&r'�
_init_weights�sb80818283������z!CLIPPreTrainedModel._init_weightsFcCst|t�r84||_dSdS�N)r}�CLIPEncoder�gradient_checkpointing)r#r��valuer&r&r'�_set_gradient_checkpointing�s8586�z/CLIPPreTrainedModel._set_gradient_checkpointingNrt)rGrHrIrJr�config_class�base_model_prefix�supports_gradient_checkpointing�_keys_to_ignore_on_load_missingr�r�r&r&r&r'rw�s5rwa=87    This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the88    library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads89    etc.)90 91    This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass.92    Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage93    and behavior.94 95    Parameters:96        config ([`CLIPConfig`]): Model configuration class with all the parameters of the model.97            Initializing with a config file does not load the weights associated with the model, only the98            configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights.99a�100    Args:101        input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):102            Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide103            it.104 105            Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and106            [`PreTrainedTokenizer.__call__`] for details.107 108            [What are input IDs?](../glossary#input-ids)109        attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):110            Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:111 112            - 1 for tokens that are **not masked**,113            - 0 for tokens that are **masked**.114 115            [What are attention masks?](../glossary#attention-mask)116        position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):117            Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0,118            config.max_position_embeddings - 1]`.119 120            [What are position IDs?](../glossary#position-ids)121        output_attentions (`bool`, *optional*):122            Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned123            tensors for more detail.124        output_hidden_states (`bool`, *optional*):125            Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for126            more detail.127        return_dict (`bool`, *optional*):128            Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.129aE130    Args:131        pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`):132            Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using133            [`AutoImageProcessor`]. See [`CLIPImageProcessor.__call__`] for details.134        output_attentions (`bool`, *optional*):135            Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned136            tensors for more detail.137        output_hidden_states (`bool`, *optional*):138            Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for139            more detail.140        return_dict (`bool`, *optional*):141            Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.142a143    Args:144        input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):145            Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide146            it.147 148            Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and149            [`PreTrainedTokenizer.__call__`] for details.150 151            [What are input IDs?](../glossary#input-ids)152        attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):153            Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:154 155            - 1 for tokens that are **not masked**,156            - 0 for tokens that are **masked**.157 158            [What are attention masks?](../glossary#attention-mask)159        position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):160            Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0,161            config.max_position_embeddings - 1]`.162 163            [What are position IDs?](../glossary#position-ids)164        pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`):165            Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using166            [`AutoImageProcessor`]. See [`CLIPImageProcessor.__call__`] for details.167        return_loss (`bool`, *optional*):168            Whether or not to return the contrastive loss.169        output_attentions (`bool`, *optional*):170            Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned171            tensors for more detail.172        output_hidden_states (`bool`, *optional*):173            Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for174            more detail.175        return_dict (`bool`, *optional*):176            Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.177csteZdZdZdef�fdd�Z					ddeejdeejdee	d	ee	d178ee	de179eeffdd
�Z
�ZS)r�z�180    Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a181    [`CLIPEncoderLayer`].182 183    Args:184        config: CLIPConfig185    rMcs:t����|_t��fdd�t�j�D��|_d|_dS)Ncsg|]}t���qSr&)rL)�.0�_�rMr&r'�186<listcomp>]sz(CLIPEncoder.__init__.<locals>.<listcomp>F)	rr rMr�187ModuleList�ranger��layersr�rgr$r�r'r Zs188 189zCLIPEncoder.__init__Nrirjrk�output_hidden_states�return_dictrlcs�dur�n|jj�|dur|n|jj}|dur|n|jj}|r"dnd}�r(dnd}|}	t|j�D]9\}190}|r<||	f}|jrU|jrU�fdd�}tj	j191�192||�|	||�}
n||	||�d�}
|
d}	�rj||
df}q1|rr||	f}|s�tdd	�|	||fD��St|	||d193�S)a�194        Args:195            inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`):196                Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation.197                This is useful if you want more control over how to convert `input_ids` indices into associated vectors198                than the model's internal embedding lookup matrix.199            attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):200                Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:201 202                - 1 for tokens that are **not masked**,203                - 0 for tokens that are **masked**.204 205                [What are attention masks?](../glossary#attention-mask)206            causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):207                Causal mask for the text model. Mask values selected in `[0, 1]`:208 209                - 1 for tokens that are **not masked**,210                - 0 for tokens that are **masked**.211 212                [What are attention masks?](../glossary#attention-mask)213            output_attentions (`bool`, *optional*):214                Whether or not to return the attentions tensors of all attention layers. See `attentions` under215                returned tensors for more detail.216            output_hidden_states (`bool`, *optional*):217                Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors218                for more detail.219            return_dict (`bool`, *optional*):220                Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.221        Nr&cs��fdd�}|S)Ncs�g|���R�Sr�r&)�inputs)r�rkr&r'�custom_forward�szJCLIPEncoder.forward.<locals>.create_custom_forward.<locals>.custom_forwardr&)r�r��rk)r�r'�create_custom_forward�sz2CLIPEncoder.forward.<locals>.create_custom_forwardr�rrcss�|]	}|dur|VqdSr�r&)r��vr&r&r'�	<genexpr>�s�z&CLIPEncoder.forward.<locals>.<genexpr>)�last_hidden_staterh�222attentions)
rMrkr��use_return_dict�	enumerater�r�r,r-�utils�223checkpoint�tupler)r#�
inputs_embedsrirjrkr�r��encoder_states�all_attentionsrh�idx�
encoder_layerr��
layer_outputsr&r�r'rF`sF&�224���225�zCLIPEncoder.forward)NNNNN)rGrHrIrJrr rr-r0rurrrrFrKr&r&r$r'r�Qs*	������226�r��input_ids_shape�dtype�device�past_key_values_lengthcCs�|\}}tj||ft�|�j|d�}tj|�d�|d�}|�||d�|�d�d�kd�|�|�}|dkrFtj	tj227||||d�|gdd�}|dddd�dd�f�|d|||�S)zB228    Make causal mask used for bi-directional self-attention.229    �r�r)rr�r�r�r*N)r-�full�finfo�minr2r1�masked_fill_�view�tor:r_�expand)r�r�r�r��bsz�tgt_len�mask�	mask_condr&r&r'�_make_causal_mask�s"230 (r�cs�eZdZdef�fdd�Zee�eeed�						dde	e231jde	e232jde	e233jd	e	ed234e	ede	ede
eeffd
d���Z�ZS)�CLIPTextTransformerrMcs@t���||_|j}t|�|_t|�|_tj	||j235d�|_dS�NrN)rr rMrSr�236embeddingsr��encoderrrVrW�final_layer_norm�r#rMrTr$r&r'r �s237238239zCLIPTextTransformer.__init__��output_typer�N�	input_idsriryrkr�r�rlc
Cs |dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td��|��}|�d|d�}|j||d�}t||j	|j240d�}	|durLt||j	�}|j|||	|||d�}241|242d}|�
|�}|tj|jd|j243d�|jtj|j244d�jdd	�f}|s�||f|245d246d�St|||247j|248jd�S)�249        Returns:250 251        NzYou have to specify input_idsr))r�ryr�)r�rirjrkr�r�rr�r*r�r��
pooler_outputrhr�)rMrkr�r��252ValueErrorr1r�r�r�r�r�rr�r�r-r2rnr�r4�argmaxr
rhr�)
r#r�riryrkr�r��input_shaperhrj�encoder_outputsr��
pooled_outputr&r&r'rF�sF��	253���zCLIPTextTransformer.forward�NNNNNN)rGrHrIrr r�CLIP_TEXT_INPUTS_DOCSTRINGrr
rr-r0rurrrFrKr&r&r$r'r��s2254�������255�r�z?The text model from CLIP without any head or projection on top.cs�eZdZeZdgZdef�fdd�Zdejfdd�Z	dd	�Z256ee�e
eed257�						ddeejd
eejdeejdeedeedeedeeeffdd���Z�ZS)�
CLIPTextModelrLrMc�"t��|�t|�|_|��dSr�)rr r��258text_model�	post_initrgr$r&r'r  �259zCLIPTextModel.__init__rlcC�260|jjjSr��r�r�r~�r#r&r&r'�get_input_embeddings&�261z"CLIPTextModel.get_input_embeddingscCs||jj_dSr�r�)r#r�r&r&r'�set_input_embeddings)sz"CLIPTextModel.set_input_embeddingsr�Nr�riryrkr�r�cCs*|dur|n|jj}|j||||||d�S)aK262        Returns:263 264        Examples:265 266        ```python267        >>> from transformers import AutoTokenizer, CLIPTextModel268 269        >>> model = CLIPTextModel.from_pretrained("openai/clip-vit-base-patch32")270        >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32")271 272        >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")273 274        >>> outputs = model(**inputs)275        >>> last_hidden_state = outputs.last_hidden_state276        >>> pooled_output = outputs.pooler_output  # pooled (EOS token) states277        ```N�r�riryrkr�r�)rMr�r�)r#r�riryrkr�r�r&r&r'rF,s�zCLIPTextModel.forwardr�)rGrHrIrr��_no_split_modulesr r�Moduler�r�rr�rr
rr-r0rurrrFrKr&r&r$r'r�s:278�������279�r�csxeZdZdef�fdd�Zee�eeed�				d
de	e280jde	ede	ed	e	ed281e
eeff282dd���Z�ZS)�CLIPVisionTransformerrMcs^t���||_|j}t|�|_t|j�|_t	j283||jd�|_t
|�|_t	j284||jd�|_dSr�)rr rMrSrr�r�force_patch_dropout�
patch_dropoutrrVrW�pre_layrnormr�r��post_layernormr�r$r&r'r Us285286287zCLIPVisionTransformer.__init__r�N�pixel_valuesrkr�r�rlcCsh|dur|n|jj}|dur|n|jj}|dur|n|jj}|dur&td��t|j�dkrH|j\}}}}}	}288}|||}|�|||	|289|�}nt|j�dkr]|j\}}
}}
}
t|d�}n	|j\}}
}
}
d}|�	|�}|�290|||�}|�|�}|j||||d�}|d}|dd�ddd�f}|�
|�}|�||d	��d�}|s�||f|dd�St|||j|jd291�S)r�Nz You have to specify pixel_values��zb c t h w -> (b t) c h wr)r�rkr�r�rr)r�)rMrkr�r�r��lenrn�reshaperr�r�r�r�r�rzr
rhr�)r#r�rkr�r��b_new�pair_newr=�bs_new�channel_new�h_new�w_newr<r�rhr�r�r�r&r&r'rF`sH
�292293�294�zCLIPVisionTransformer.forward�NNNN)rGrHrIrr r�CLIP_VISION_INPUTS_DOCSTRINGrr
rr-rvrurrrFrKr&r&r$r'r�Ts&295�����296�r�zAThe vision model from CLIP without any head or projection on top.cs�eZdZeZdZdef�fdd�Zdejfdd�Z	e297e�ee
ed�								ddeejd298eedeedeedeee
ff299d
d���Z�ZS)�CLIPVisionModelr�rMcr�r�)rr r��vision_modelr�rgr$r&r'r �r�zCLIPVisionModel.__init__rlcCr�r�)rr�r�r�r&r&r'r��r�z$CLIPVisionModel.get_input_embeddingsr�Nrkr�r�cCs&|dur|n|jj}|j||||d�S)a�300        Returns:301 302        Examples:303 304        ```python305        >>> from PIL import Image306        >>> import requests307        >>> from transformers import AutoProcessor, CLIPVisionModel308 309        >>> model = CLIPVisionModel.from_pretrained("openai/clip-vit-base-patch32")310        >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32")311 312        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"313        >>> image = Image.open(requests.get(url, stream=True).raw)314 315        >>> inputs = processor(images=image, return_tensors="pt")316 317        >>> outputs = model(**inputs)318        >>> last_hidden_state = outputs.last_hidden_state319        >>> pooled_output = outputs.pooler_output  # pooled CLS states320        ```N�r�rkr�r�)rMr�r)r#r�rkr�r�r&r&r'rF�s�zCLIPVisionModel.forwardr)rGrHrIrr��main_input_namer rr�r�rrrr
rr-rvrurrrFrKr&r&r$r'r�s,321�����322�rcsZeZdZeZdef�fdd�Zdd�Zdd�Zee	�						dd	e323ejd324e325ejde326ejde327e
d
e328e
de329e
dejfdd��Zee�				dde330ejde331e
d
e332e
de333e
dejf334dd��Zee�eeed�								dd	e335ejde336ejd337e338ejde339ejde340e
de341e
d
e342e
de343e
deeeffdd���Z�ZS)r�rMcst��|�t|jt�stdt|j��d���t|jt�s(tdt|j��d���|j}|j}|j	|_	|j344|_345|j|_|j|_|j
|_
|j|_|j|_t|�|_t|�|_tj|j|j
dd�|_tj|j|j
dd�|_t�t�|jj��|_|��|� �|�!|jj"|�dS)NzKconfig.text_config is expected to be of type CLIPTextConfig but is of type �.zOconfig.vision_config is expected to be of type CLIPVisionConfig but is of type F)r�)#rr r}�text_configrr��type�
vision_configrr[�lora_r�346lora_alpha�lora_dropout�projection_dimrSr�r�r�r�r�rrr�r�r�r^r-�tensorrM�logit_scale_init_value�logit_scaler��convert_to_lora�347resize_posr�)r#rMr348rr$r&r'r �s<����349350zLanguageBindDepth.__init__cCs`|jdkrdS|jrgd�}ngd�}t|j|j||jdgd�}d|jj_t|jj|�|j_dS)Nr)ztemporal_attn.k_projztemporal_attn.v_projztemporal_attn.q_projztemporal_attn.out_projztemporal_mlp.fc1ztemporal_mlp.fc2)r�r�r�r��none)�rr�target_modulesrr��modules_to_saveF)	r
r[rrrrr��is_gradient_checkpointingr)r#rrMr&r&r'rs351352�353z!LanguageBindDepth.convert_to_loracCs�|jdkr|jdkr|j|jg|_t|jt�r|j|jgn|j|j_|j��}|d}|j}|jjd|j	|jjd|j	g}d}|d|d|}||j354dkrUdS|d|d|_|jd|_|�
dt�|j��d��t�|j|j�}	|r�|d|�||d�}355}nd|}356}tt�t|���gd}|�d|d|dd��dddd�}tj||d	d357dd�}|�dddd��d|d|dd�d}|358dur�tj|359|gdd
�}
n|}
|
�|�|d<|	|_|j�|�dS)Nrrrry)rr)r{r)��bicubicTF)r1�mode�	antialias�
align_cornersr*)�num_mel_bins�
target_length�360image_sizer}r4rMr��361state_dictr��362patch_sizern�num_patches�
num_positions�register_bufferr-r2r�r�	EmbeddingrT�math�sqrtr�r��permute�F�interpolater:r��load_state_dict)r#�mr�old_pos_embed_state_dict�
old_pos_embedr��	grid_size�extra_tokens�new_seq_len�new_position_embedding�pos_emb_tok�pos_emb_img�
old_grid_size�
new_pos_embedr&r&r'rsD"363$364$�*zLanguageBindDepth.resize_posNr�riryrkr�r�rlc365Csh|dur|n|jj}|dur|n|jj}|dur|n|jj}|j||||||d�}|d}|�|�}	|	S)a�366        Returns:367            text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by368            applying the projection layer to the pooled output of [`CLIPTextModel`].369 370        Examples:371 372        ```python373        >>> from transformers import AutoTokenizer, CLIPModel374 375        >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")376        >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32")377 378        >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")379        >>> text_features = model.get_text_features(**inputs)380        ```Nr�r)rMrkr�r�r�r�)381r#r�riryrkr�r��text_outputsr��
text_featuresr&r&r'�get_text_featuresKs��	382z#LanguageBindDepth.get_text_featuresr�cCsd|dur|n|jj}|dur|n|jj}|dur|n|jj}|j||||d�}|d}|�|�}|S)aD383        Returns:384            image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by385            applying the projection layer to the pooled output of [`CLIPVisionModel`].386 387        Examples:388 389        ```python390        >>> from PIL import Image391        >>> import requests392        >>> from transformers import AutoProcessor, CLIPModel393 394        >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")395        >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32")396 397        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"398        >>> image = Image.open(requests.get(url, stream=True).raw)399 400        >>> inputs = processor(images=image, return_tensors="pt")401 402        >>> image_features = model.get_image_features(**inputs)403        ```Nrr)rMrkr�r�rr�)r#r�rkr�r��vision_outputsr��image_featuresr&r&r'�get_image_featureszs��404z$LanguageBindDepth.get_image_featuresr��return_lossc		Cs&|dur|n|jj}|dur|n|jj}|dur|n|jj}|j||||d�}	|j||||||d�}405|	d}|�|�}|406d}|�|�}||jdddd�}||jdddd�}|j	�407�}
t�||�
��|
}|�
�}d}|rst|�}|s�|||||408|	f}|dur�|f|S|St||||||409|	d	�S)410a�411        Returns:412 413        Examples:414 415        ```python416        >>> from PIL import Image417        >>> import requests418        >>> from transformers import AutoProcessor, CLIPModel419 420        >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")421        >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32")422 423        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"424        >>> image = Image.open(requests.get(url, stream=True).raw)425 426        >>> inputs = processor(427        ...     text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True428        ... )429 430        >>> outputs = model(**inputs)431        >>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score432        >>> probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities433        ```Nrr�rr{r)T)�pr+�keepdim)�loss�logits_per_image�logits_per_text�text_embeds�image_embeds�text_model_output�vision_model_output)rMrkr�r�rr�r�r��normr�expr-�matmulr]rr)r#r�r�riryr@rkr�r�r=r:rGrFrrErDrC�outputr&r&r'rF�sT&���	434435436�zLanguageBindDepth.forwardr�r)NNNNNNNN)rGrHrIrr�r rrrr�rr-r0rurvr<rr?�CLIP_INPUTS_DOCSTRINGrr�437LongTensorrrrFrKr&r&r$r'r��s�&0��������.������0438��������	�439 440�r�)r)9r)�typingrrrr-�einopsr�peftrrr�torch.nnr	r,�transformersr441r�transformers.modeling_outputsrr
Z&transformers.models.clip.modeling_cliprrrrrrrrr�transformers.utilsrr�configuration_depthrrrr�rrLrw�CLIP_START_DOCSTRINGr�rrNr��Sizer�r�r4r�r�r�r�rr�r&r&r&r'�<module>sX,.gE &i����442�M�9O�6