XaviXva/Video-LLaVA
0
1o
2 ��Ne4 � @ s> d dl Z d dlmZ d dlmZmZmZ G dd� dej�ZdS )� N)�ViTMAEForPreTraining�3AutoConfig�AutoImageProcessorc s� e Zd Zd� fdd� Zdd� Zdd� Ze�� d d4� �Ze d d � �Z5e d
d� �Z e dd� �Z e dd� �Z
e dd� �Ze dd� �Z� ZS )�MAEVisionTower� ./cache_dirFc sZ t � �� d| _|| _|| _|j| _t|dd�| _|s!| � � d S t6j | j| jd�| _ d S )NF�mm_vision_select_feature�patch�� cache_dir)
�super�__init__� is_loadedr7 �vision_tower_name�mm_vision_select_layer� select_layer�getattr�select_feature�8load_modelr �from_pretrained�cfg_only)�self� vision_tower�argsr9 �10delay_load�� __class__� �N/scc_cephfs/yy/lb/LLaVA-Video-YY/llava/model/multimodal_encoder/mae_encoder.pyr s 11 zMAEVisionTower.__init__c C sD t j| j| jd�| _tj| j| jd�}|j| _| j�d� d| _ d S )Nr FT)12r r r r13 �image_processorr �vitr �requires_grad_r
)r r r r r r s14 15zMAEVisionTower.load_modelc C sP |j | j }| jdkr|d d �dd �f }|S | jdkr |}|S td| j� ���)Nr � � cls_patchzUnexpected select feature: )�
hidden_statesr r �16ValueError)r �image_forward_outs�image_featuresr r r �feature_select s 1718��z MAEVisionTower.feature_selectc C s� t |�tu r.g }|D ]!}| j|j| j| jd��d�dd�}| �|��|j�}|�|� q19|S | j|j| j| jd�dd�}| �|��|j�}|S )N��device�dtyper T)�output_hidden_states) �type�listr �tor) r* � unsqueezer' �append)r �imagesr& �image�image_forward_out�
image_featurer% r r r �forward) s " �zMAEVisionTower.forwardc C s t jd| j| j| jd�S )Nr! r( )�torch�zeros� hidden_sizer) r* �r r r r �
dummy_feature7 s z MAEVisionTower.dummy_featurec C � | j jS �N)r r* r9 r r r r* ; � zMAEVisionTower.dtypec C r; r< )r r) r9 r r r r) ? r= zMAEVisionTower.devicec C s | j r| jjS | jS r< )r
r �configr r9 r r r r> C s zMAEVisionTower.configc C r; r< )r> r8 r9 r r r r8 J r= zMAEVisionTower.hidden_sizec C s | j j| j j d S )N� )r> �20image_size�21patch_sizer9 r r r � num_patchesN s zMAEVisionTower.num_patches)r F)�__name__�22__module__� __qualname__r r r' r6 �no_gradr5 �propertyr: r* r) r> r8 rB �
__classcell__r r r r r s$ 23
2425262728r ) r6 �torch.nn�nn� transformersr r r �Moduler r r r r �<module> s