XaviXva/Video-LLaVA
0
1o
2 p�6e�3 � @ s> d dl Z d dlmZ d dlmZmZmZ G dd� dej�ZdS )� N)�CLIPVisionModel�CLIPImageProcessor�CLIPVisionConfigc s� e Zd Zd� fdd� Zdd� Zdd� Ze�� dd � �Ze d4d � �Z5e d d
� �Z e dd� �Z e dd� �Z
e dd� �Ze dd� �Z� ZS )�CLIPVisionTowerFc sN t � �� d| _|| _|j| _t|dd�| _|s | �� d S t �6| j�| _ d S )NF�mm_vision_select_feature�patch) �super�__init__� is_loaded�vision_tower_name�mm_vision_select_layer� select_layer�getattr�select_feature�7load_modelr �from_pretrained�cfg_only)�self� vision_tower�args�8delay_load�� __class__� �O/scc_cephfs/yy/lb/LLaVA-Video-YY/llava/model/multimodal_encoder/clip_encoder.pyr s 9 zCLIPVisionTower.__init__c C s2 t �| j�| _t�| j�| _| j�d� d| _d S )NFT)r r r Zimage_processorr r �requires_grad_r10 �r r r r r s 11zCLIPVisionTower.load_modelc C sP |j | j }| jdkr|d d �dd �f }|S | jdkr |}|S td| j� ���)Nr � Z cls_patchzUnexpected select feature: )�
hidden_statesr
r �12ValueError)r �image_forward_outs�image_featuresr r r �feature_select s 1314��z CLIPVisionTower.feature_selectc C s� t |�tu r.g }|D ]!}| j|j| j| jd��d�dd�}| �|��|j�}|�|� q15|S | j|j| j| jd�dd�}| �|��|j�}|S )N��device�dtyper T)�output_hidden_states) �type�listr �tor$ r% � unsqueezer" �append)r �imagesr! �imageZimage_forward_outZ
image_featurer r r r �forward' s " �zCLIPVisionTower.forwardc C s t jd| j| j| jd�S )Nr r# )�torch�zeros� hidden_sizer$ r% r r r r �
dummy_feature5 s z CLIPVisionTower.dummy_featurec C � | j jS �N)r r% r r r r r% 9 � zCLIPVisionTower.dtypec C r3 r4 )r r$ r r r r r$ = r5 zCLIPVisionTower.devicec C s | j r| jjS | jS r4 )r16 r �configr r r r r r6 A s zCLIPVisionTower.configc C r3 r4 )r6 r1 r r r r r1 H r5 zCLIPVisionTower.hidden_sizec C s | j j| j j d S )N� )r6 Z17image_sizeZ18patch_sizer r r r � num_patchesL s zCLIPVisionTower.num_patches)F)�__name__�19__module__� __qualname__r r r" r/ �no_gradr. �propertyr2 r% r$ r6 r1 r8 �
__classcell__r r r r r s$ 20 21
2223242526r ) r/ �torch.nn�nn� transformersr r r �Moduler r r r r �<module> s