XaviXva/Video-LLaVA
0
1o
2 � e� � @ s� d dl Z d dlZd dlmZ d dlmZ d dlmZ d dlmZm Z d dl3m Z dZ dZ
d d4� Zd d � ZG d
d� dej�Zdd� Zdd� ZG dd� de�ZdS )� N)�Image)�nn)�5transforms)�ProcessorMixin�
BatchEncoding)� BatchFeature)g3<�4'��?gwg�� M�?gy{�� �?)g��B�91�?g�w�t.��?g�� U��?c C s t | t�s| gS | S �N)�6isinstance�list)�x� r �ND:\LLaVA\llava\model\multimodal_encoder\languagebind\depth\processing_depth.py�make_list_of_images s 7r c C s t �| t j��d�S )N�float32)�cv2�imread�IMREAD_UNCHANGED�astype)�pathr r r
�
opencv_loader s r c s* e Zd Z d� fdd� Zdd� Z� ZS )� DepthNormr �{�G�z�?c s t � �� || _|| _d| _d S )Ng @�@)�super�__init__� max_depth� min_depth�scale)�selfr r �� __class__r r
r s 89zDepthNorm.__init__c C sp || j }|j| jd�}| jdkr |j| jd�}|| j }n||�� }t�|��d��ddd�}|� t�10� �S )N)�minr )�max� � ) r �clipr r r! �torch�11from_numpy� unsqueeze�repeat�to�get_default_dtype)r �imageZ depth_imgr r r
�forward s 1213 zDepthNorm.forward)r r )�__name__�14__module__� __qualname__r r, �
__classcell__r r r r
r s15 �16r c C s@ | j } t�t| jd�tjdtjjd�t�d�t� t17t �g�}|S )N)r �� )�
interpolation) �
vision_configr �Composer r �Resize�InterpolationMode�BICUBIC�18CenterCrop� Normalize�OPENAI_DATASET_MEAN�OPENAI_DATASET_STD)�config� transformr r r
�get_depth_transform, s 1920�� r> c C s t | �}||�}|S r )r )Z21depth_pathr= �depthZ
depth_outputsr r r
�load_and_transform_depth; s r@ c sD e Zd Zg ZdZd
� fdd� Zddd�Zdd d22�Zdd d �Z� Z S )�LanguageBindDepthProcessor�LanguageBindDepthTokenizerNc s2 t � jdi |�� || _t|�| _t| _|| _d S )Nr )r r r< r> r= r@ �image_processor� tokenizer)r r< rD �kwargsr r r
r D s23 2425z#LanguageBindDepthProcessor.__init__�M c s� |d u r |d u r t d��|d ur � j|f|dd|d�|��}|d ur4t|�}� fdd�|D �}t�|�}|d urB|d urB||d< |S |d urH|S d|iS )Nz?You have to specify either text or images. Both cannot be none.�26max_lengthT)rG �padding�27truncation�return_tensorsc s g | ] }� � |� j��qS r )rC r= )�.0r+ �r r r
�28<listcomp>U s z7LanguageBindDepthProcessor.__call__.<locals>.<listcomp>� pixel_values)�29ValueErrorrD r r% �stack)r �images�text�context_lengthrJ rE �encoding�image_featuresr rL r
�__call__K s$ ��30z#LanguageBindDepthProcessor.__call__Tc O � | j j|d|i|��S )z�31 This method forwards all its arguments to CLIPTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please32 refer to the docstring of this method for more information.33 �skip_special_tokens)rD � batch_decode�r rX �argsrE r r r
rY ` � z'LanguageBindDepthProcessor.batch_decodec O rW )z�34 This method forwards all its arguments to CLIPTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to35 the docstring of this method for more information.36 rX )rD �decoderZ r r r
r] g r\ z!LanguageBindDepthProcessor.decoder )NNrF N)T)37r- r. r/ �38attributes�tokenizer_classr rV rY r] r0 r r r r
rA @ s 3940rA )r r% �PILr r � torchvisionr � transformersr r Z#transformers.image_processing_utilsr r: r; r r �Moduler r> r@ rA r r r r
�<module> s