XaviXva/Video-LLaVA
0
1o
2 5�De� � @ s� d dl Z d dlZd dlZd dlmZmZ ej�d� d dlZd dl m3Z4 d dl m Z d dl
mZmZ d dlmZ d dlmZ d d lmZmZmZ d d5lmZmZmZm Z d d l m Z m Z m Z d Z!d
Z"dd� Z#dd� Z$ ddd�Z%G dd� de�Z&dS )� N)� VideoReader�cpu�torch)�Image)�6transforms)�ProcessorMixin�
BatchEncoding)� BatchFeature)� EncodedVideo)�Compose�Lambda�ToTensor)�NormalizeVideo�RandomCropVideo�RandomHorizontalFlipVideo�CenterCropVideo)�ApplyTransformToKey�ShortSideScale�UniformTemporalSubsample)g3<�4'��?gwg�� M�?gy{�� �?)g��B�91�?g�w�t.��?g�� U��?c C s t | t�s| gS | S �N)�7isinstance�list��x� r �f/scc_cephfs/yy/lb/LLaVA-Video-YY/llava/model/multimodal_encoder/languagebind/video/processing_video.py�make_list_of_images s 8r c C s� | j } | jdkr+tdtt| j�tdd� �ttt d�t9dd�t d�t dd �g�d10�}|S | jd krKttd d� �ttt d�t11dd�t d�t dd �g�}|S | jd
krkttdd� �ttt d�t12dd�t d�t dd �g�}|S t
d��)N� pytorchvideo�videoc S � | d S �Ng �o@r r r r r �<lambda>" � z%get_video_transform.<locals>.<lambda>)�mean�std�� )�sizeg �?)�p)�key� transform�decordc S r r r r r r r r! 0 r" �opencvc S r r r r r r r r! < r" �Evideo_decode_backend should specify in (pytorchvideo, decord, opencv))�
vision_config�video_decode_backendr r r �13num_framesr r �OPENAI_DATASET_MEAN�OPENAI_DATASET_STDr r r � NameError)�configr) r r r �get_video_transform sH 141516���(17�1819��20�2122�� �r4 r+ � � c C sv |dkr&t j| ddd�}|j}|}|d ur|n|} |j|| d�}23||24�} | S |dkrZtj�d� t| td�d�} t | �}t25j d|d |t d26�}
| �
|
�}27|28�d dd d �}29||30�} | S |d
kr�t�| �}t |�tj��}t31j d|d |t d32�}
g }33|
D ]+}|�d |� |�� \}}|s�td| � ���t�|tj�}|34�t�|��d dd �� qz|�� tj |35d d�}36||37�} | S t d��)Nr r* F)�decoderZ decode_audio)� start_sec�end_secr r )�ctx� )�dtype� � r+ � zvideo error at )�dimr, ) r38 � from_path�duration�get_clipr* �bridge�39set_bridger r �len�np�linspace�int� get_batch�permute�cv2Z VideoCapture�getZCAP_PROP_FRAME_COUNT�set�read�40ValueErrorZcvtColorZ
COLOR_BGR2RGB�appendr �41from_numpy�release�stackr2 )�42video_pathr) r. Zclip_start_secZ clip_end_secr/ r rB r8 r9 �43video_dataZ
video_outputsZ decord_vrZ
frame_id_listZcv2_vrZ frame_idx�ret�framer r r �load_and_transform_videoH sD � 44�45 �rY c sL e Zd Zg ZdZd� fdd� Zddd�Zdd � Zdd d �Zdd
d�Z � Z46S )�LanguageBindVideoProcessor�LanguageBindVideoTokenizerNc s2 t � jdi |�� || _t|�| _t| _|| _d S )Nr )�super�__init__r3 r4 r) rY �image_processor� tokenizer)�selfr3 r_ �kwargs�� __class__r r r] z s47 4849z#LanguageBindVideoProcessor.__init__�M c s� |d u r |d u r t d��|d ur � j|f|dd|d�|��}|d ur4t|�}� fdd�|D �}t�|�}|d urB|d urB||d< |S |d urH|S d|iS )Nz?You have to specify either text or images. Both cannot be none.�50max_lengthT)re �padding�51truncation�return_tensorsc s, g | ]}� j |� j� jjj� jjjd ��qS ))r. r/ )r^ r) r3 r- r. r/ )�.0�image�r` r r �52<listcomp>� s 53�54�z7LanguageBindVideoProcessor.__call__.<locals>.<listcomp>� pixel_values)rP r_ r r rT )r` �images�text�context_lengthrh ra �encoding�image_featuresr rk r �__call__� s( ��55�56z#LanguageBindVideoProcessor.__call__c C s | j ||d�S )N)rn rh )rs )r` rn rh r r r �57preprocess� s z%LanguageBindVideoProcessor.preprocessTc O � | j j|d|i|��S )z�58 This method forwards all its arguments to CLIPTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please59 refer to the docstring of this method for more information.60 �skip_special_tokens)r_ � batch_decode�r` rv �argsra r r r rw � � z'LanguageBindVideoProcessor.batch_decodec O ru )z�61 This method forwards all its arguments to CLIPTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to62 the docstring of this method for more information.63 rv )r_ �decoderx r r r r{ � rz z!LanguageBindVideoProcessor.decoder )NNrd N)T) �__name__�64__module__� __qualname__�65attributes�tokenizer_classr] rs rt rw r{ �
__classcell__r r rb r rZ v s 66 67rZ )r+ r5 Nr6 )'r rL r* r r rD rE �numpyrG �PILr � torchvisionr � transformersr r Z#transformers.image_processing_utilsr Z pytorchvideo.data.encoded_videor68 Ztorchvision.transformsr r r
Z(torchvision.transforms._transforms_videor r r r Zpytorchvideo.transformsr r r r0 r1 r r4 rY rZ r r r r �<module> s0 169�.