XaviXva/Video-LLaVA
0
1o
2 ،8eγ � @ s� d dl Z d dlmZmZmZ d dlZd dlmZ d dlm Z m3Z4 d dlm Z d dl m
Z d dlmZmZ d dlmZmZ d d lmZmZmZmZmZmZm Z m Z m Z d d5l m Z m!Z! d d l"m#Z#m$Z$m%Z% G d
d� de j&�Z'G dd� de j&�Z(G dd� de�Z)dZ*dZ+dZ,dZ-G dd� de j&�Z. d+dej/dej0dej1d e2fd d �Z3G d d � d e j&�Z4ed!e*�G d"d#� d#e)��Z5G d$d%� d%e j&�Z6ed&e*�G d'd(� d(e)��Z7ee*�G d)d*� d*e)��Z8dS ),� N)�Optional�Tuple�Union�� rearrange)�6LoraConfig�get_peft_model)�nn)�7functional)�PreTrainedModel�add_start_docstrings)�BaseModelOutput�BaseModelOutputWithPooling) �CLIPMLP�
CLIPAttention�CLIPTextEmbeddings�CLIPVisionEmbeddings� CLIPVisionModelWithProjection�CLIPTextModelWithProjection� _expand_mask�8CLIPOutput� clip_loss)�%add_start_docstrings_to_model_forward�replace_return_docstrings� )�LanguageBindVideoConfig�CLIPVisionConfig�CLIPTextConfigc s* e Zd ZdZd� fdd� Zdd� Z� ZS )� PatchDropoutz*9 https://arxiv.org/abs/2212.0079410 Tc s6 t � �� d| krdk sJ � J �|| _|| _d S )Nr � �?)�super�__init__�prob�exclude_first_token)�selfr" r# �� __class__� �d/scc_cephfs/yy/lb/LLaVA-Video-YY/llava/model/multimodal_encoder/languagebind/video/modeling_video.pyr! s 11 12zPatchDropout.__init__c C s4 | j r| jdkr13|S | jr#|d d �d d�f |d d �dd �f }}ntj�tj|d d �d d�f �}|�� d }|�� d }t�|�}|d }d| j }t dt14|| ��} |dkrit� ||�}15|16j | dd�j
} n t� ||�}17|18j | dd�j
} | �d��d|d�} t| d�} ||| f }| jr�tj||fdd�}|S )N� r r ).N�������dimzb t n -> (b t) n)�trainingr" r# �torch�jit�annotate�Tensor�size�arange�max�int�randn�topk�indices� unsqueeze�repeatr �cat) r$ �x�B�T�19cls_tokens�batch�20num_tokens�
batch_indices� keep_prob�num_patches_keep�rand�patch_indices_keepr' r' r( �forward s, , 2122 23 zPatchDropout.forward)T)�__name__�24__module__� __qualname__�__doc__r! rG �
__classcell__r' r' r% r( r s r c sT e Zd Zdef� fdd� Z d dejdejdejdee d e ej25 f26d27d �Z � Z S )
�CLIPEncoderLayer�configc s� t � �� |j| _t|�| _tj| j|jd�| _ t28|�| _ tj| j|jd�| _ |j
| _
| j
r`|j| _t�t�d|j|j��| _tjj| j|jd d� |j| _t|�| _tj| j|jd�| _d S d S )N��epsr � �std)r r! � hidden_size� embed_dimr � self_attnr � LayerNorm�layer_norm_eps� layer_norm1r �mlp� layer_norm2�
add_time_attn�29num_frames�t� Parameterr. �zeros�temporal_embedding�init�normal_�
temporal_attn�temporal_layer_norm1�r$ rN r% r' r( r! B s 30313233�zCLIPEncoderLayer.__init__F�
hidden_states�attention_mask�causal_attention_mask�output_attentions�returnc C s | j rU|j\}}}| j}|dkr3|jd }t|d|d�}|| jdd�d|�dd�f }t|d|d�}|} t|d|d�}| �|�}| j||||d�\}}34| t|d|d� }|} | �|�}| j||||d�\}}35| | }|} | � |�}| �36|�}| | }|f} |r�| |37f7 } | S )aI 38 Args:39 hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)`40 attention_mask (`torch.FloatTensor`): attention mask of size41 `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.42 `(config.encoder_attention_heads,)`.43 output_attentions (`bool`, *optional*):44 Whether or not to return the attentions tensors of all attention layers. See `attentions` under45 returned tensors for more detail.46 r z(b t) n d -> (b n) t d)r^ Nz(b n) t d -> (b t) n d)�n)rg rh ri rj ) r\ �shaper^ r ra re rd rY rV r[ rZ ) r$ rg rh ri rj �btrl �dr^ �residual� attn_weights�outputsr' r' r( rG V sF 47 4849�50 5152�535455zCLIPEncoderLayer.forward�F)
rH rI rJ r r! r. r1 r �boolr � FloatTensorrG rL r' r' r% r( rM A s ������rM c @ s4 e Zd ZdZeZdZdZdgZdd� Z d dd �Z56d57S ) �CLIPPreTrainedModelz�58 An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained59 models.60 �clipT� position_idsc C s� | j j}t|t�r#|jjjjd|d d� |jjjjd|d d� �nt|t �rX| j j}t61j j|j d|j
d | d� t62j j|jj|j j| d� t63j j|jj|j j| d� n�t|t�r�| j j}|j
d d|j j d | }|j
d | }t64j j|jj|d� t65j j|jj|d� t66j j|jj|d� t67j j|jj|d� n�t|t�r�| j j}|j jd d|j j d | }d|j j d | }t68j j|jj|d� t69j j|jj|d� nZt|t��rt70j j|jj|j d | j j d� t71j j|j j|j d | j j d� n1t|t ��rt72j j|j j| j jd | j j d� nt|t ��r2t73j j|jj| j jd | j j d� t|t74j!��rF|j"j�#� |jj�$d� t|t75j%��r[|j"du�r]|j"j�#� dS dS dS ) zInitialize the weightsr) g{�G�z�?)�meanrS rQ rR � r N)&rN �initializer_factor�76isinstancer �token_embedding�weight�datarc �position_embeddingr r rb �class_embeddingrU �patch_embedding�initializer_ranger �num_hidden_layers�q_proj�k_proj�v_proj�out_projr rT �fc1�fc2�LanguageBindVideo�text_projection�text_embed_dim�visual_projection�vision_embed_dimr r rW �bias�zero_�fill_�Linear)r$ �module�factor� in_proj_std� out_proj_std�fc_stdr' r' r( �
_init_weights� sb 77 78 79 80 � �� � � �z!CLIPPreTrainedModel._init_weightsFc C s t |t�r81||_d S d S �N)r| � CLIPEncoder�gradient_checkpointing)r$ r� �valuer' r' r( �_set_gradient_checkpointing� s 8283�z/CLIPPreTrainedModel._set_gradient_checkpointingNrs ) rH rI rJ rK r � config_class�base_model_prefix� supports_gradient_checkpointing� _keys_to_ignore_on_load_missingr� r� r' r' r' r( rv � s 5rv a= 84 This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the85 library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads86 etc.)87 88 This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass.89 Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage90 and behavior.91 92 Parameters:93 config ([`CLIPConfig`]): Model configuration class with all the parameters of the model.94 Initializing with a config file does not load the weights associated with the model, only the95 configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights.96a� 97 Args:98 input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):99 Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide100 it.101 102 Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and103 [`PreTrainedTokenizer.__call__`] for details.104 105 [What are input IDs?](../glossary#input-ids)106 attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):107 Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:108 109 - 1 for tokens that are **not masked**,110 - 0 for tokens that are **masked**.111 112 [What are attention masks?](../glossary#attention-mask)113 position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):114 Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0,115 config.max_position_embeddings - 1]`.116 117 [What are position IDs?](../glossary#position-ids)118 output_attentions (`bool`, *optional*):119 Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned120 tensors for more detail.121 output_hidden_states (`bool`, *optional*):122 Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for123 more detail.124 return_dict (`bool`, *optional*):125 Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.126aE 127 Args:128 pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`):129 Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using130 [`AutoImageProcessor`]. See [`CLIPImageProcessor.__call__`] for details.131 output_attentions (`bool`, *optional*):132 Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned133 tensors for more detail.134 output_hidden_states (`bool`, *optional*):135 Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for136 more detail.137 return_dict (`bool`, *optional*):138 Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.139a 140 Args:141 input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):142 Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide143 it.144 145 Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and146 [`PreTrainedTokenizer.__call__`] for details.147 148 [What are input IDs?](../glossary#input-ids)149 attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):150 Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:151 152 - 1 for tokens that are **not masked**,153 - 0 for tokens that are **masked**.154 155 [What are attention masks?](../glossary#attention-mask)156 position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):157 Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0,158 config.max_position_embeddings - 1]`.159 160 [What are position IDs?](../glossary#position-ids)161 pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`):162 Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using163 [`AutoImageProcessor`]. See [`CLIPImageProcessor.__call__`] for details.164 return_loss (`bool`, *optional*):165 Whether or not to return the contrastive loss.166 output_attentions (`bool`, *optional*):167 Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned168 tensors for more detail.169 output_hidden_states (`bool`, *optional*):170 Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for171 more detail.172 return_dict (`bool`, *optional*):173 Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.174c st e Zd ZdZdef� fdd� Z ddeej deej dee d ee d175ee d e176e e f f d d
�Z
� ZS )r� z�177 Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a178 [`CLIPEncoderLayer`].179 180 Args:181 config: CLIPConfig182 rN c s: t � �� � | _t�� fdd�t� j�D ��| _d| _d S )Nc s g | ]}t � ��qS r' )rM )�.0�_�rN r' r( �183<listcomp>] s z(CLIPEncoder.__init__.<locals>.<listcomp>F) r r! rN r �184ModuleList�ranger� �layersr� rf r% r� r( r! Z s 185 186zCLIPEncoder.__init__Nrh ri rj �output_hidden_states� return_dictrk c s � dur� n| j j� |dur|n| j j}|dur|n| j j}|r"dnd}� r(dnd}|} t| j�D ]9\}187} |r<|| f }| jrU| jrU� fdd�} tj j188�189| | �| ||�}
n| | ||� d�}
|
d } � rj||
d f }q1|rr|| f }|s�t dd � | ||fD ��S t | ||d190�S ) a� 191 Args:192 inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`):193 Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation.194 This is useful if you want more control over how to convert `input_ids` indices into associated vectors195 than the model's internal embedding lookup matrix.196 attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):197 Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:198 199 - 1 for tokens that are **not masked**,200 - 0 for tokens that are **masked**.201 202 [What are attention masks?](../glossary#attention-mask)203 causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):204 Causal mask for the text model. Mask values selected in `[0, 1]`:205 206 - 1 for tokens that are **not masked**,207 - 0 for tokens that are **masked**.208 209 [What are attention masks?](../glossary#attention-mask)210 output_attentions (`bool`, *optional*):211 Whether or not to return the attentions tensors of all attention layers. See `attentions` under212 returned tensors for more detail.213 output_hidden_states (`bool`, *optional*):214 Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors215 for more detail.216 return_dict (`bool`, *optional*):217 Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.218 Nr' c s � �fdd�}|S )Nc s � g | ���R � S r� r' )�inputs)r� rj r' r( �custom_forward� s zJCLIPEncoder.forward.<locals>.create_custom_forward.<locals>.custom_forwardr' )r� r� �rj )r� r( �create_custom_forward� s z2CLIPEncoder.forward.<locals>.create_custom_forwardr� r r c s s � | ] }|d ur|V qd S r� r' )r� �vr' r' r( � <genexpr>� s � z&CLIPEncoder.forward.<locals>.<genexpr>)�last_hidden_staterg �219attentions)
rN rj r� �use_return_dict� enumerater� r� r- r. �utils�220checkpoint�tupler
)r$ �
inputs_embedsrh ri rj r� r� �encoder_states�all_attentionsrg �idx�
encoder_layerr� �
layer_outputsr' r� r( rG ` sF &� 221 ���222�zCLIPEncoder.forward)NNNNN)rH rI rJ rK r r! r r. r1 rt r r r
rG rL r' r' r% r( r� Q s* ������223�r� �input_ids_shape�dtype�device�past_key_values_lengthc C s� | \}}t j||ft �|�j|d�}t j|�d�|d�}|�||d �|�d�d�k d� |�|�}|dkrFt j t j224||||d�|gdd�}|dddd�dd�f � |d||| �S )zB225 Make causal mask used for bi-directional self-attention.226 �r� r* r r �r� r� r+ N) r. �full�finfo�minr3 r2 � masked_fill_�view�tor; r` �expand)r� r� r� r� �bsz�tgt_len�mask� mask_condr' r' r( �_make_causal_mask� s "227 (r� c s� e Zd Zdef� fdd� Zee�eeed� dde e228j de e229j de e230j d e e d231e e d e e d e
eef fd
d���Z� ZS )�CLIPTextTransformerrN c s@ t � �� || _|j}t|�| _t|�| _tj ||j232d�| _ d S �NrO ) r r! rN rT r �233embeddingsr� �encoderr rW rX �final_layer_norm�r$ rN rU r% r' r( r! � s 234235236z CLIPTextTransformer.__init__�� output_typer� N� input_idsrh rx rj r� r� rk c
C s |dur|n| j j}|dur|n| j j}|dur|n| j j}|du r&td��|�� }|�d|d �}| j||d�}t||j |j237d�} |durLt ||j �}| j ||| |||d�}238|239d } | �
| �} | tj| jd | j240d�|jtj| j241d�jdd �f } |s�| | f|242d243d� S t| | |244j|245jd �S ) �246 Returns:247 248 Nz You have to specify input_idsr* )r� rx r� )r� rh ri rj r� r� r r� r+ r �r� �
pooler_outputrg r� )rN rj r� r� �249ValueErrorr2 r� r� r� r� r� r r� r� r. r3 rm r� r5 �argmaxr rg r� )
r$ r� rh rx rj r� r� � input_shaperg ri �encoder_outputsr� �
pooled_outputr' r' r( rG � sF � � 250���zCLIPTextTransformer.forward�NNNNNN)rH rI rJ r r! r �CLIP_TEXT_INPUTS_DOCSTRINGr r r r. r1 rt r r rG rL r' r' r% r( r� � s2 251�������252�r� z?The text model from CLIP without any head or projection on top.c s� e Zd ZeZdgZdef� fdd� Zdejfdd�Z dd � Z253e e �e
eed254� dd eej d
eej deej dee dee dee deeef fdd���Z� ZS )�
CLIPTextModelrM rN c �" t � �|� t|�| _| �� d S r� )r r! r� �255text_model� post_initrf r% r' r( r! � 256 zCLIPTextModel.__init__rk c C �257 | j jjS r� �r� r� r} �r$ r' r' r( �get_input_embeddings& � 258z"CLIPTextModel.get_input_embeddingsc C s || j j_d S r� r� )r$ r� r' r' r( �set_input_embeddings) s z"CLIPTextModel.set_input_embeddingsr� Nr� rh rx rj r� r� c C s* |dur|n| j j}| j||||||d�S )aK 259 Returns:260 261 Examples:262 263 ```python264 >>> from transformers import AutoTokenizer, CLIPTextModel265 266 >>> model = CLIPTextModel.from_pretrained("openai/clip-vit-base-patch32")267 >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32")268 269 >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")270 271 >>> outputs = model(**inputs)272 >>> last_hidden_state = outputs.last_hidden_state273 >>> pooled_output = outputs.pooler_output # pooled (EOS token) states274 ```N�r� rh rx rj r� r� )rN r� r� )r$ r� rh rx rj r� r� r' r' r( rG , s �zCLIPTextModel.forwardr� )rH rI rJ r r� �_no_split_modulesr! r �Moduler� r� r r� r r r r. r1 rt r r rG rL r' r' r% r( r� s: 275�������276�r� c sx e Zd Zdef� fdd� Zee�eeed� d
de e277j de e de e d e e d278e
eef f279d d ���Z� ZS )�CLIPVisionTransformerrN c s^ t � �� || _|j}t|�| _t|j�| _t j280||j d�| _ t
|�| _t j281||j d�| _d S r� )r r! rN rT r r� r �force_patch_dropout�
patch_dropoutr rW rX � pre_layrnormr� r� �post_layernormr� r% r' r( r! U s 282283 284z CLIPVisionTransformer.__init__r� N� pixel_valuesrj r� r� rk c s~ |dur|n| j j}|dur|n| j j}|dur|n| j j}|du r&td��t|j�dkrH|j\}}}}} }285} || | � |�� | | |286| �}n t|j�dkr]|j\� } }} } t|d�}n |j\� } } } d}| � |�}
| �287|
� |�}
| � |
�}
| j |
|||d�}|d }|dd�ddd�f }| �
|�}|�� |d ��d�}� fd288d �|jD �|_|s�||f|dd� S t|||j|jd �S )
r� Nz You have to specify pixel_values� � zb c t h w -> (b t) c h wr )r� rj r� r� r r* c s g | ] }t |d � d��qS )z(b t) n c -> b t n c)�br )r� �i�r= r' r( r� � s z1CLIPVisionTransformer.forward.<locals>.<listcomp>r� )rN rj r� r� r� �lenrm �reshaper r� r� r� r� r� ry rg r r� )r$ r� rj r� r� �b_new�pair_newr> �bs_new� channel_new�h_new�w_newr� rg r� r� r� r' r� r( rG ` sJ
� 289290�291�z CLIPVisionTransformer.forward�NNNN)rH rI rJ r r! r � CLIP_VISION_INPUTS_DOCSTRINGr r r r. ru rt r r rG rL r' r' r% r( r� T s& 292�����293�r� zAThe vision model from CLIP without any head or projection on top.c s� e Zd ZeZdZdef� fdd� Zdejfdd�Z e294e �e e
ed� ddeej d295ee d ee d ee deee
f f296d
d���Z� ZS )�CLIPVisionModelr� rN c r� r� )r r! r� � vision_modelr� rf r% r' r( r! � r� zCLIPVisionModel.__init__rk c C r� r� )r r� r� r� r' r' r( r� � r� z$CLIPVisionModel.get_input_embeddingsr� Nrj r� r� c C s& |dur|n| j j}| j||||d�S )a� 297 Returns:298 299 Examples:300 301 ```python302 >>> from PIL import Image303 >>> import requests304 >>> from transformers import AutoProcessor, CLIPVisionModel305 306 >>> model = CLIPVisionModel.from_pretrained("openai/clip-vit-base-patch32")307 >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32")308 309 >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"310 >>> image = Image.open(requests.get(url, stream=True).raw)311 312 >>> inputs = processor(images=image, return_tensors="pt")313 314 >>> outputs = model(**inputs)315 >>> last_hidden_state = outputs.last_hidden_state316 >>> pooled_output = outputs.pooler_output # pooled CLS states317 ```N�r� rj r� r� )rN r� r )r$ r� rj r� r� r' r' r( rG � s �zCLIPVisionModel.forwardr )rH rI rJ r r� �main_input_namer! r r� r� r r r r r r. ru rt r r rG rL r' r' r% r( r � s, 318�����319�r c sZ e Zd ZeZdef� fdd� Zdd� Zdd� Zee � dd e320e j d321e322e j d e323e j d e324e
d
e325e
de326e
de jfdd��Zee� dde327e j d e328e
d
e329e
de330e
de jf331dd��Zee�eeed� dd e332e j de333e j d334e335e j d e336e j de337e
d e338e
d
e339e
de340e
deeef fdd���Z� ZS ) r� rN c s� t � �|� t|jt�stdt|j�� d���t|jt�s(tdt|j�� d���|j}|j}|j | _ |j341| _342|j | _ |j | _ |j
| _
|j| _|j| _t|�| _t|�| _tj| j| j
dd�| _tj| j| j
dd�| _t�t�| j j ��| _ | � � d S )NzKconfig.text_config is expected to be of type CLIPTextConfig but is of type �.zOconfig.vision_config is expected to be of type CLIPVisionConfig but is of type F)r� ) r r! r| � text_configr r� �type�
vision_configr r\ �lora_r�343lora_alpha� lora_dropout�projection_dimrT r� r� r� r� r� r r r� r� r� r_ r. �tensorrN �logit_scale_init_value� logit_scaler� )r$ rN r r r% r' r( r! � s8 �� ��344345 zLanguageBindVideo.__init__c C s` | j dkrd S | jrg d�}ng d�}t| j | j|| jdg d�}d| jj_t| jj|�| j_d S )Nr )ztemporal_attn.k_projztemporal_attn.v_projztemporal_attn.q_projztemporal_attn.out_projztemporal_mlp.fc1ztemporal_mlp.fc2)r� r� r� r� �none)�rr �target_modulesr r� �modules_to_saveF) r r\ r r r r r� �is_gradient_checkpointingr )r$ r rN r' r' r( �convert_to_lora346 s 347348�349z!LanguageBindVideo.convert_to_lorac C s� |j dkr|jdkr|j |jg|_t|jt�r |j|jgn|j|j_|j�� }|d }|j}|jjd |j |jjd |j g}d}|d |d | }||j350d krUd S |d |d |_ |j d |_ |�
dt�|j ��d�� t�|j |j�} |r�|d |� ||d � }351} nd |}352} tt�t| ���gd } | �d| d | d d��dddd�} tj| |d d353d d �} | �dddd��d|d |d d�d } |354d ur�tj|355| gdd
�}
n| }
|
� |�|d< | |_|j� |� d S )Nr r~ r rx )r r* rz r* � �bicubicTF)r2 �mode� antialias�
align_cornersr+ ) � num_mel_bins�
target_length�356image_sizer| r5 rN r� �357state_dictr� �358patch_sizerm � num_patches�
num_positions�register_bufferr. r3 r� r � EmbeddingrU �math�sqrtr� r� �permute�F� interpolater; r� �load_state_dict)r$ �mr �old_pos_embed_state_dict�
old_pos_embedr� � grid_size� extra_tokens� new_seq_len�new_position_embedding� pos_emb_tok� pos_emb_img�
old_grid_size�
new_pos_embedr' r' r( �359resize_pos sD "360$ 361$�*z LanguageBindVideo.resize_posNr� rh rx rj r� r� rk c 362 C sh |dur|n| j j}|dur|n| j j}|dur|n| j j}| j||||||d�}|d }| �|�} | S )a� 363 Returns:364 text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by365 applying the projection layer to the pooled output of [`CLIPTextModel`].366 367 Examples:368 369 ```python370 >>> from transformers import AutoTokenizer, CLIPModel371 372 >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")373 >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32")374 375 >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")376 >>> text_features = model.get_text_features(**inputs)377 ```Nr� r )rN rj r� r� r� r� )378r$ r� rh rx rj r� r� � text_outputsr� �
text_featuresr' r' r( �get_text_featuresN s �� 379z#LanguageBindVideo.get_text_featuresr� c C sd |dur|n| j j}|dur|n| j j}|dur|n| j j}| j||||d�}|d }| �|�}|S )aD 380 Returns:381 image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by382 applying the projection layer to the pooled output of [`CLIPVisionModel`].383 384 Examples:385 386 ```python387 >>> from PIL import Image388 >>> import requests389 >>> from transformers import AutoProcessor, CLIPModel390 391 >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")392 >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32")393 394 >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"395 >>> image = Image.open(requests.get(url, stream=True).raw)396 397 >>> inputs = processor(images=image, return_tensors="pt")398 399 >>> image_features = model.get_image_features(**inputs)400 ```Nr r )rN rj r� r� r r� )r$ r� rj r� r� �vision_outputsr� �image_featuresr' r' r( �get_image_features} s ��401z$LanguageBindVideo.get_image_featuresr� � return_lossc C s& |dur|n| j j}|dur|n| j j}|dur|n| j j}| j||||d�} | j||||||d�}402| d } | �| �} |403d } | �| �} | | jdddd� } | | jdddd� } | j �404� }
t � | | �
� �|
}|�
� }d}|rst|�}|s�||| | |405| f}|dur�|f| S |S t|||| | |406| d �S )407a� 408 Returns:409 410 Examples:411 412 ```python413 >>> from PIL import Image414 >>> import requests415 >>> from transformers import AutoProcessor, CLIPModel416 417 >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")418 >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32")419 420 >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"421 >>> image = Image.open(requests.get(url, stream=True).raw)422 423 >>> inputs = processor(424 ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True425 ... )426 427 >>> outputs = model(**inputs)428 >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score429 >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities430 ```Nr r� r rz r* T)�pr, �keepdim)�loss�logits_per_image�logits_per_text� text_embeds� image_embeds�text_model_output�vision_model_output)rN rj r� r� r r� r� r� �normr �expr. �matmulr^ r r )r$ r� r� rh rx rB rj r� r� r? r< rI rH r rG rF rE �outputr' r' r( rG � sT &��� 431432433�zLanguageBindVideo.forwardr� r )NNNNNNNN)rH rI rJ r r� r! r r; r r� r r. r1 rt ru r> r rA �CLIP_INPUTS_DOCSTRINGr r �434LongTensorr r rG rL r' r' r% r( r� � s� &0������� �.����� �0435�������� �436 437�r� )r )9r* �typingr r r r. �einopsr �peftr r r �torch.nnr438 r- � transformersr r � transformers.modeling_outputsr
r Z&transformers.models.clip.modeling_clipr r r r r r r r r �transformers.utilsr r �configuration_videor r r r� r rM rv �CLIP_START_DOCSTRINGr� r rP r� �Sizer� r� r5 r� r� r� r� r r� r' r' r' r( �<module> sX ,.gE &i����439�M�9R�6