Apppublic
Aluode/PerceptionLabPortable

sourceHugging Faceupdated 9mo agoView on Hugging Face
0likes
rootpython_embedLibsite-packagestransformersmodelsowlv2__pycache__modeling_owlv2.cpython-310.pyc
modeling_owlv2.cpython-310.pyc656 linesDownload Raw Back to __pycache__
1o

2/�Yi�0�@s�dZddlmZddlmZddlmZmZmZddl	Z	ddl	m3Z4mZddlm
Z
dd	lmZmZdd5lmZddlmZmZddlmZdd
lmZmZmZmZmZmZddlm Z m!Z!m"Z"e�riddl#m$Z$e�%e&�Z'de	j6de	j7fdd�Z(de	j8de	j9fdd�Z)eeGdd�de���Z*de10de11fdd�Z+de12de13fdd�Z,d d!�Z-d"d#�Z.eed$d%�Gd&d'�d'e���Z/eed(d%�Gd)d*�d*e���Z0Gd+d,�d,ej1�Z2Gd-d.�d.ej1�Z3Gd/d0�d0ej1�Z4Gd1d2�d2ej1�Z5Gd3d4�d4e�Z6eGd5d6�d6e��Z7Gd7d8�d8ej1�Z8Gd9d:�d:ej1�Z9Gd;d<�d<e7�Z:Gd=d>�d>ej1�Z;Gd?d@�d@e7�Z<eGdAdB�dBe7��Z=GdCdD�dDej1�Z>GdEdF�dFej1�Z?GdGdH�dHe7�Z@gdI�ZAdS)JzPyTorch OWLv2 model.�)�	dataclass)�	lru_cache)�Any�Optional�UnionN)�Tensor�nn�)�ACT2FN)� _create_4d_causal_attention_mask�_prepare_4d_attention_mask)�GradientCheckpointingLayer)�BaseModelOutput�BaseModelOutputWithPooling)�PreTrainedModel)�ModelOutput�auto_docstring�filter_out_non_signature_kwargs�is_vision_available�logging�	torch_int�)�Owlv2Config�Owlv2TextConfig�Owlv2VisionConfig)�center_to_corners_format�logits�returncCstj�|tjt|�|jd��S)N��device)r�14functionalZ
cross_entropy�torch�arange�lenr)r�r$��E:\DocsHouse\542 percep lab latest\PerceptionLab\PerceptionLab_Portable\python_embed\Lib\site-packages\transformers/models/owlv2/modeling_owlv2.py�contrastive_loss3sr&�15similaritycCs t|�}t|���}||dS)Ng@)r&�t)r'Zcaption_lossZ16image_lossr$r$r%�17owlv2_loss8sr)c@s�eZdZUdZdZeejed<dZ	eejed<dZ18eejed<dZeejed<dZeejed<dZ
eed<dZeed	<d19eefdd�ZdS)
�Owlv2Outputa�20    loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`):21        Contrastive loss for image-text similarity.22    logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`):23        The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text24        similarity scores.25    logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`):26        The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image27        similarity scores.28    text_embeds (`torch.FloatTensor` of shape `(batch_size * num_max_text_queries, output_dim`):29        The text embeddings obtained by applying the projection layer to the pooled output of [`Owlv2TextModel`].30    image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`):31        The image embeddings obtained by applying the projection layer to the pooled output of32        [`Owlv2VisionModel`].33    text_model_output (tuple[`BaseModelOutputWithPooling`]):34        The output of the [`Owlv2TextModel`].35    vision_model_output (`BaseModelOutputWithPooling`):36        The output of the [`Owlv2VisionModel`].37    N�loss�logits_per_image�logits_per_text�text_embeds�image_embeds�text_model_output�vision_model_outputrc�t�fdd����D��S)Nc3�.�|]}|dvr�|nt�|���VqdS�)r0r1N��getattr�to_tuple��.0�k��selfr$r%�	<genexpr>^�38��39�z'Owlv2Output.to_tuple.<locals>.<genexpr>��tuple�keysr;r$r;r%r7]��zOwlv2Output.to_tuple)�__name__�40__module__�__qualname__�__doc__r+rr!�FloatTensor�__annotations__r,r-r.r/r0rr1r@rr7r$r$r$r%r*>s41r*r(cCsD|��r|jtjtjfvr|S|��S|jtjtjfvr|S|��S�N)	Zis_floating_point�dtyper!�float32Zfloat64�floatZint32Zint64�int)r(r$r$r%�_upcastesrN�boxescCsHt|�}|dd�df|dd�df|dd�df|dd�dfS)a�42    Computes the area of a set of bounding boxes, which are specified by its (x1, y1, x2, y2) coordinates.43 44    Args:45        boxes (`torch.FloatTensor` of shape `(number_of_boxes, 4)`):46            Boxes for which the area will be computed. They are expected to be in (x1, y1, x2, y2) format with `0 <= x147            < x2` and `0 <= y1 < y2`.48 49    Returns:50        `torch.FloatTensor`: a tensor containing the area for each box.51    N�rr	r)rN)rOr$r$r%�box_areans@rQc52Cs�t|�}t|�}t�|dd�ddd�f|dd�dd�f�}t�|dd�ddd�f|dd�dd�f�}||jdd�}|dd�dd�df|dd�dd�df}|dd�df||}||}	|	|fS)NrPr��minr)rQr!�maxrS�clamp)53�boxes1�boxes2Zarea1Zarea2Zleft_topZright_bottom�width_height�inter�union�iour$r$r%�box_ious..,r\cCs*|dd�dd�f|dd�dd�fk��std|����|dd�dd�f|dd�dd�fk��s:td|����t||�\}}t�|dd�ddd�f|dd�dd�f�}t�|dd�ddd�f|dd�dd�f�}||jdd�}|dd�dd�df|dd�dd�df}||||S)z�54    Generalized IoU from https://giou.stanford.edu/. The boxes should be in [x0, y0, x1, y1] (corner) format.55 56    Returns:57        `torch.FloatTensor`: a [N, M] pairwise matrix, where N = len(boxes1) and M = len(boxes2)58    NrPz<boxes1 must be in [x0, y0, x1, y1] (corner) format, but got z<boxes2 must be in [x0, y0, x1, y1] (corner) format, but got rrRr)�all�59ValueErrorr\r!rSrTrU)rVrWr[rZ�top_left�bottom_rightrXZarear$r$r%�generalized_box_iou�s,	,..,raz560    Output type of [`Owlv2ForObjectDetection`].61    )Zcustom_introc@s�eZdZUdZdZeejed<dZ	ee62ed<dZeejed<dZeejed<dZ
eejed<dZeejed<dZeejed	<dZeejed63<dZeed<dZeed<d
eefdd�ZdS)�Owlv2ObjectDetectionOutputa�64    loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` are provided)):65        Total loss as a linear combination of a negative log-likehood (cross-entropy) for class prediction and a66        bounding box loss. The latter is defined as a linear combination of the L1 loss and the generalized67        scale-invariant IoU loss.68    loss_dict (`Dict`, *optional*):69        A dictionary containing the individual losses. Useful for logging.70    logits (`torch.FloatTensor` of shape `(batch_size, num_patches, num_queries)`):71        Classification logits (including no-object) for all queries.72    objectness_logits (`torch.FloatTensor` of shape `(batch_size, num_patches, 1)`):73        The objectness logits of all image patches. OWL-ViT represents images as a set of image patches where the74        total number of patches is (image_size / patch_size)**2.75    pred_boxes (`torch.FloatTensor` of shape `(batch_size, num_patches, 4)`):76        Normalized boxes coordinates for all queries, represented as (center_x, center_y, width, height). These77        values are normalized in [0, 1], relative to the size of each individual image in the batch (disregarding78        possible padding). You can use [`~Owlv2ImageProcessor.post_process_object_detection`] to retrieve the79        unnormalized bounding boxes.80    text_embeds (`torch.FloatTensor` of shape `(batch_size, num_max_text_queries, output_dim`):81        The text embeddings obtained by applying the projection layer to the pooled output of [`Owlv2TextModel`].82    image_embeds (`torch.FloatTensor` of shape `(batch_size, patch_size, patch_size, output_dim`):83        Pooled output of [`Owlv2VisionModel`]. OWLv2 represents images as a set of image patches and computes image84        embeddings for each patch.85    class_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`):86        Class embeddings of all image patches. OWLv2 represents images as a set of image patches where the total87        number of patches is (image_size / patch_size)**2.88    text_model_output (tuple[`BaseModelOutputWithPooling`]):89        The output of the [`Owlv2TextModel`].90    vision_model_output (`BaseModelOutputWithPooling`):91        The output of the [`Owlv2VisionModel`].92    Nr+�	loss_dictr�objectness_logits�93pred_boxesr.r/�class_embedsr0r1rcr2)Nc3r3r4r5r8r;r$r%r=�r>z6Owlv2ObjectDetectionOutput.to_tuple.<locals>.<genexpr>r?r;r$r;r%r7�rBz#Owlv2ObjectDetectionOutput.to_tuple)rCrDrErFr+rr!rGrHrc�dictrrdrer.r/rfr0rr1r@rr7r$r$r$r%rb�s94rbzL95    Output type of [`Owlv2ForObjectDetection.image_guided_detection`].96    c@s�eZdZUdZdZeejed<dZ	eejed<dZ97eejed<dZeejed<dZeejed<dZ
eejed<dZeed	<dZeed98<deefdd
�ZdS)�%Owlv2ImageGuidedObjectDetectionOutputa99    logits (`torch.FloatTensor` of shape `(batch_size, num_patches, num_queries)`):100        Classification logits (including no-object) for all queries.101    image_embeds (`torch.FloatTensor` of shape `(batch_size, patch_size, patch_size, output_dim`):102        Pooled output of [`Owlv2VisionModel`]. OWLv2 represents images as a set of image patches and computes103        image embeddings for each patch.104    query_image_embeds (`torch.FloatTensor` of shape `(batch_size, patch_size, patch_size, output_dim`):105        Pooled output of [`Owlv2VisionModel`]. OWLv2 represents images as a set of image patches and computes106        image embeddings for each patch.107    target_pred_boxes (`torch.FloatTensor` of shape `(batch_size, num_patches, 4)`):108        Normalized boxes coordinates for all queries, represented as (center_x, center_y, width, height). These109        values are normalized in [0, 1], relative to the size of each individual target image in the batch110        (disregarding possible padding). You can use [`~Owlv2ImageProcessor.post_process_object_detection`] to111        retrieve the unnormalized bounding boxes.112    query_pred_boxes (`torch.FloatTensor` of shape `(batch_size, num_patches, 4)`):113        Normalized boxes coordinates for all queries, represented as (center_x, center_y, width, height). These114        values are normalized in [0, 1], relative to the size of each individual query image in the batch115        (disregarding possible padding). You can use [`~Owlv2ImageProcessor.post_process_object_detection`] to116        retrieve the unnormalized bounding boxes.117    class_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`):118        Class embeddings of all image patches. OWLv2 represents images as a set of image patches where the total119        number of patches is (image_size / patch_size)**2.120    text_model_output (tuple[`BaseModelOutputWithPooling`]):121        The output of the [`Owlv2TextModel`].122    vision_model_output (`BaseModelOutputWithPooling`):123        The output of the [`Owlv2VisionModel`].124    Nrr/�query_image_embeds�target_pred_boxes�query_pred_boxesrfr0r1rcr2)Nc3r3r4r5r8r;r$r%r=r>zAOwlv2ImageGuidedObjectDetectionOutput.to_tuple.<locals>.<genexpr>r?r;r$r;r%r7
rBz.Owlv2ImageGuidedObjectDetectionOutput.to_tuple)rCrDrErFrrr!rGrHr/rirjrkrfr0rr1r@rr7r$r$r$r%rh�s125rhcs\eZdZdef�fdd�Zdejdededejfdd	�Zddej	de126dejfd
d�Z�ZS)�Owlv2VisionEmbeddings�configcs�t���|j|_||_|j|_t�t�	|j��|_127tj|j|j|j|jdd�|_
|j|jd|_|jd|_t�|j|j�|_|jdt�|j��d�dd�dS)NF)Zin_channelsZout_channelsZkernel_sizeZstride�biasrPr�position_ids�r�������128persistent)�super�__init__�129patch_sizerm�hidden_size�	embed_dimr�	Parameterr!Zrandn�class_embeddingZConv2dZnum_channels�patch_embedding�130image_size�num_patches�
num_positions�	Embedding�position_embedding�register_bufferr"�expand�r<rm��	__class__r$r%rus 131�"zOwlv2VisionEmbeddings.__init__�132embeddings�height�widthrc
Cs|jdd}|jj�d�}|jdd}tj��s(||kr(||kr(|�|j�S|dd�dd�f}|dd�dd�f}|jd}	||j}133||j}t	|d�}|�134d|||	�}|�dddd�}tj
j||135|fdd	d136�}|�dddd��dd|	�}tj||fdd�S)a137        This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution138        images. This method is also adapted to support torch.jit tracing.139 140        Adapted from:141        - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and142        - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211143        rrNrqg�?r	rPZbicubicF)�size�modeZ
align_corners��dim)�shaper��weight�	unsqueezer!Zjit�144is_tracingrorvr�reshapeZpermuterr Zinterpolate�view�cat)
r<r�r�r�r}r�r~Zclass_pos_embedZpatch_pos_embedr�Z145new_heightZ	new_widthZsqrt_num_positionsr$r$r%�interpolate_pos_encoding+s*146147148149�z.Owlv2VisionEmbeddings.interpolate_pos_encodingF�pixel_valuesr�c150Csz|j\}}}}|�|�}|�d��dd�}|j�|dd�}tj||gdd�}	|r3|	|�|	||�}	|	S|	|�	|j151�}	|	S)NrPrrqr�)r�r{�flatten�	transposerzr�r!r�r�r�ro)152r<r�r��153batch_size�_r�r�Zpatch_embedsrfr�r$r$r%�forwardTs154�zOwlv2VisionEmbeddings.forward�F)
rCrDrErrur!rrMr�rG�boolr��
__classcell__r$r$r�r%rls$)rlc	sXeZdZdef�fdd�Z			ddeejdeejdeejdej	fd	d155�Z156�ZS)�Owlv2TextEmbeddingsrmcsPt���t�|j|j�|_t�|j|j�|_|j	dt157�|j��d�dd�dS)NrorpFrr)
rtrurrZ158vocab_sizerw�token_embeddingZmax_position_embeddingsr�r�r!r"r�r�r�r$r%ruds159160�zOwlv2TextEmbeddings.__init__N�	input_idsro�
inputs_embedsrcCsb|dur	|jdn|jd}|dur|jdd�d|�f}|dur&|�|�}|�|�}||}|S)Nrq�����)r�ror�r�)r<r�ror�Z161seq_lengthZposition_embeddingsr�r$r$r%r�ns162163zOwlv2TextEmbeddings.forward)NNN)rCrDrErrurr!�164LongTensorrGrr�r�r$r$r�r%r�cs�����r�cs�eZdZdZ�fdd�Zdejdedefdd�Z					165ddejde	ejd
e	ejde	e166deeje	eje	eejff167dd�Z�Z
S)�Owlv2Attentionz=Multi-headed attention from 'Attention Is All You Need' papercs�t���||_|j|_|j|_|j|j|_|j|j|jkr-td|j�d|j�d���|jd|_	|j168|_t�
|j|j�|_t�
|j|j�|_t�
|j|j�|_t�
|j|j�|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).��)rtrurmrwrxZnum_attention_heads�	num_heads�head_dimr^�scaleZattention_dropout�dropoutr�Linear�k_proj�v_proj�q_proj�out_projr�r�r$r%ru�s"169170��zOwlv2Attention.__init__�tensor�seq_len�bszcCs |�|||j|j��dd���S)NrrP)r�r�r�r��171contiguous)r<r�r�r�r$r$r%�_shape�s zOwlv2Attention._shapeNF�
hidden_states�attention_mask�causal_attention_mask�output_attentionsrcCs�|��\}}}|�|�|j}|�|�|�d|�}	|�|�|�d|�}172||jd|jf}|�|||�j|�}|	j|�}	|173j|�}174|	�d�}t	�175||	�dd��}
|
��||j||fkrmtd||j||f�d|
������|dur�|��|d||fkr�td|d||f�d|������|
�||j||�|}
|
�||j||�}
|dur�|��|d||fkr�td|d||f�d|������|
�||j||�|}
|
�||j||�}
t
jj|
dd�}
|r�|
�||j||�}|�||j||�}
nd}t
jj|
|j|jd	�}|�|176j�}t	�177||178�}|��||j||jfk�r*td179||j||jf�d|������|�||j||j�}|�dd�}|�|||�}|�|�}||fS)z#Input shape: Batch x Time x ChannelrqrrPz$Attention weights should be of size z	, but is Nz!Attention mask should be of size r�)�p�trainingz `attn_output` should be of size )r�r�r�r�r�r�r�r�r�r!Zbmmr�r^rr Zsoftmaxr�r��torJr�r�)r<r�r�r�r�r�Ztgt_lenrxZquery_statesZ180key_statesZvalue_statesZ181proj_shapeZsrc_len�attn_weightsZattn_weights_reshapedZ182attn_probsZattn_outputr$r$r%r��sf	183184185�������186zOwlv2Attention.forward�NNF)rCrDrErFrur!rrMr�rr�r@r�r�r$r$r�r%r��s$������r�cs2eZdZ�fdd�Zdejdejfdd�Z�ZS)�Owlv2MLPcsDt���||_t|j|_t�|j|j	�|_187t�|j	|j�|_dSrI)rtrurmr188Z189hidden_act�
activation_fnrr�rwZintermediate_size�fc1�fc2r�r�r$r%ru�s190191zOwlv2MLP.__init__r�rcCs"|�|�}|�|�}|�|�}|SrI)r�r�r�)r<r�r$r$r%r��s192193194zOwlv2MLP.forward)rCrDrErur!rr�r�r$r$r�r%r��sr�csTeZdZdef�fdd�Z	ddejdejdejdeed	e	ej195f196d197d�Z�ZS)
�Owlv2EncoderLayerrmcsRt���|j|_t|�|_tj|j|jd�|_	t198|�|_tj|j|jd�|_dS�N��eps)
rtrurwrxr��	self_attnr�	LayerNorm�layer_norm_eps�layer_norm1r��mlp�layer_norm2r�r�r$r%rus199200201zOwlv2EncoderLayer.__init__Fr�r�r�r�rcCsd|}|�|�}|j||||d�\}}||}|}|�|�}|�|�}||}|f}|r0||f7}|S)aI202        Args:203            hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)`204            attention_mask (`torch.FloatTensor`): attention mask of size205                `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.206                `(config.encoder_attention_heads,)`.207            output_attentions (`bool`, *optional*):208                Whether or not to return the attentions tensors of all attention layers. See `attentions` under209                returned tensors for more detail.210        )r�r�r�r�)r�r�r�r�)r<r�r�r�r�Zresidualr��outputsr$r$r%r�s"211212�213214215zOwlv2EncoderLayer.forwardr�)
rCrDrErrur!rrr�r@rGr�r�r$r$r�r%r��s
������r�c@s4eZdZUeed<dZdZdgZdej	fdd�Z216dS)	�Owlv2PreTrainedModelrm�owlv2Tr��modulecCsZ|jj}t|t�r"|jjjjd|dd�|jjjjd|dd�n�t|t	�rSt217jj|jd|j
d|d�t218jj|jj|jj|d�t219jj|jj|jj|d�n�t|t�r�|j
dd|jjd|}|j
d|}t220jj|jj|d�t221jj|jj|d�t222jj|jj|d�t223jj|jj|d�n`t|t�r�|jjdd|jjd|}d|jjd|}t224jj|jj|d�t225jj|jj|d�n,t|t�r�t226jj|jj|jd|d�t227jj|jj|jd|d�|jj� |jj!�t|t228j"��r|j#j�$�|jj� d�t|t229j%��r)|jjjd|d�|j#du�r+|j#j�$�dSdSdS)	zInitialize the weights�g{�G�z�?)�mean�stdr�)r�rP��?N)&rmZinitializer_factor�230isinstancer�r�r��dataZnormal_r�rlr�initrzrxr{Zinitializer_ranger��num_hidden_layersr�r�r�r�r�rwr�r��231Owlv2Model�text_projection�text_embed_dim�visual_projection�vision_embed_dim�logit_scaleZfill_�logit_scale_init_valuer�rnZzero_r�)r<r��factorZin_proj_stdZout_proj_stdZfc_stdr$r$r%�
_init_weights9sN232233234235 236���z"Owlv2PreTrainedModel._init_weightsN)rCrDrErrHZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesr�Moduler�r$r$r$r%r�1s237r�csteZdZdZdef�fdd�Z					ddeejdeejdee	d	ee	d238ee	de239eeffdd
�Z
�ZS)�Owlv2Encoderz�240    Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a241    [`Owlv2EncoderLayer`].242 243    Args:244        config: Owlv2Config245    rmcs4t���t��fdd�t�j�D��|_d|_dS)Ncsg|]}t���qSr$)r�)r9r��rmr$r%�246<listcomp>nsz)Owlv2Encoder.__init__.<locals>.<listcomp>F)rtrurZ247ModuleList�ranger��layersZgradient_checkpointingr�r�r�r%ruls248 249zOwlv2Encoder.__init__Nr�r�r��output_hidden_states�return_dictrcCs�|dur|n|jj}|dur|n|jj}|dur|n|jj}|r"dnd}|r(dnd}|}	|jD]}250|r8||	f}|251|	|||d�}|d}	|rM||df}q/|rU||	f}|sctdd�|	||fD��St|	||d�S)	a�252        Args:253            inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`).254            attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):255                Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:256                - 1 for tokens that are **not masked**,257                - 0 for tokens that are **masked**.258                [What are attention masks?](../glossary#attention-mask)259            causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):260                Causal mask for the text model. Mask values selected in `[0, 1]`:261                - 1 for tokens that are **not masked**,262                - 0 for tokens that are **masked**.263                [What are attention masks?](../glossary#attention-mask)264            output_attentions (`bool`, *optional*):265                Whether or not to return the attentions tensors of all attention layers. See `attentions` under266                returned tensors for more detail.267            output_hidden_states (`bool`, *optional*):268                Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors269                for more detail.270            return_dict (`bool`, *optional*):271                Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.272        Nr$)r�rrcs��|]	}|dur|VqdSrIr$)r9�vr$r$r%r=���z'Owlv2Encoder.forward.<locals>.<genexpr>)�last_hidden_stater��273attentions)rmr�r��use_return_dictr�r@r)r<r�r�r�r�r�r�Zencoder_statesZall_attentionsr�Z
encoder_layerZ
layer_outputsr$r$r%r�qs6�274275��276�zOwlv2Encoder.forward�NNNNN)rCrDrErFrrurr!rr�rr@rr�r�r$r$r�r%r�cs*������277�r�cszeZdZdef�fdd�Ze					ddejdeejdeejdee	d	ee	d278ee	de279eeffdd
��Z
�ZS)�Owlv2TextTransformerrmcs@t���||_|j}t|�|_t|�|_tj	||j280d�|_dSr�)rtrurmrwr�r�r��encoderrr�r��final_layer_norm)r<rmrxr�r$r%ru�s281282283zOwlv2TextTransformer.__init__Nr�r�ror�r�r�rc
Cs|dur|n|jj}|dur|n|jj}|dur|n|jj}|��}|�d|d�}|j||d�}t||j|j	d�}	|durDt284||j�}|j|||	|||d�}285|286d}|�|�}|t
j|jd|j	d�|�t
j�jdd��|j	�f}|s||f|287dd�St|||288j|289jd	�S)290a|291        input_ids (`torch.LongTensor` of shape `(batch_size * num_max_text_queries, sequence_length)`):292            Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See293            [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input294            IDs?](../glossary#input-ids)295        Nrq)r�ror)r�r�r�r�r�r�rr�r�r��
pooler_outputr�r�)rmr�r�r�r�r�r�rrJrrr�r�r!r"r�r�rMZargmaxrr�r�)
r<r�r�ror�r�r�Zinput_shaper�r��encoder_outputsr��
pooled_outputr$r$r%r��sF�296��	297���zOwlv2TextTransformer.forwardr�)rCrDrErrurr!rrr�rr@rr�r�r$r$r�r%r��s.�������298�r�cs�eZdZUeed<def�fdd�Zdejfdd�Zdd�Z	e299								dd300ejde
ejde
ed
e
ede
edeeeffdd��Z�ZS)�Owlv2TextModelrmc�"t��|�t|�|_|��dSrI)rtrur��301text_model�	post_initr�r�r$r%ru�302zOwlv2TextModel.__init__rcC�303|jjjSrI�r�r�r�r;r$r$r%�get_input_embeddings304�305z#Owlv2TextModel.get_input_embeddingscCs||jj_dSrIr�)r<�valuer$r$r%�set_input_embeddings
sz#Owlv2TextModel.set_input_embeddingsNr�r�r�r�r�cC�|j|||||d�S)a�306        input_ids (`torch.LongTensor` of shape `(batch_size * num_max_text_queries, sequence_length)`):307            Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See308            [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input309            IDs?](../glossary#input-ids)310 311        Examples:312        ```python313        >>> from transformers import AutoProcessor, Owlv2TextModel314 315        >>> model = Owlv2TextModel.from_pretrained("google/owlv2-base-patch16")316        >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16")317        >>> inputs = processor(318        ...     text=[["a photo of a cat", "a photo of a dog"], ["photo of a astranaut"]], return_tensors="pt"319        ... )320        >>> outputs = model(**inputs)321        >>> last_hidden_state = outputs.last_hidden_state322        >>> pooled_output = outputs.pooler_output  # pooled (EOS token) states323        ```�r�r�r�r�r�)r�)r<r�r�r�r�r�r$r$r%r�s�zOwlv2TextModel.forward)NNNN)rCrDrErrHrurr�r�rrr!rrr�rr@rr�r�r$r$r�r%r�s.324������325�r�csleZdZdef�fdd�Ze				ddejdee	dee	d	ee	d326ee	de327eeffdd
��Z
�ZS)�Owlv2VisionTransformerrmcsPt���||_t|�|_tj|j|jd�|_	t328|�|_tj|j|jd�|_dSr�)
rtrurmrlr�rr�rwr��
pre_layernormr�r��post_layernormr�r�r$r%ru9s329330331zOwlv2VisionTransformer.__init__NFr�r�r�r�r�rcCs�|dur|n|jj}|dur|n|jj}|dur|n|jj}|jjjj}|�|�}|j||d�}|�	|�}|j332||||d�}|d}	|	dd�ddd�f}333|�|334�}335|s^|	|336f|dd�St|	|337|j
|jd�S)N)r�)r�r�r�r�rrr�)rmr�r�r�r�r{r�rJr�rr�rrr�r�)r<r�r�r�r�r�Zexpected_input_dtyper�r�r�r�r$r$r%r�Bs2	�338339�340�zOwlv2VisionTransformer.forward)NNFN)rCrDrErrurr!rGrr�rr@rr�r�r$r$r�r%r8s(	������341�rcs�eZdZUeed<dZdef�fdd�Zdejfdd�Z	e342						ddeej
d343eedeeded
eedeeeffdd��Z�ZS)�Owlv2VisionModelrmr�cr�rI)rtrur�vision_modelr�r�r�r$r%rutr�zOwlv2VisionModel.__init__rcCr�rI)rr�r{r;r$r$r%r�zr�z%Owlv2VisionModel.get_input_embeddingsNFr�r�r�r�cCr)a�344        Examples:345        ```python346        >>> from PIL import Image347        >>> import requests348        >>> from transformers import AutoProcessor, Owlv2VisionModel349 350        >>> model = Owlv2VisionModel.from_pretrained("google/owlv2-base-patch16")351        >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16")352        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"353        >>> image = Image.open(requests.get(url, stream=True).raw)354 355        >>> inputs = processor(images=image, return_tensors="pt")356 357        >>> outputs = model(**inputs)358        >>> last_hidden_state = outputs.last_hidden_state359        >>> pooled_output = outputs.pooler_output  # pooled CLS states360        ```�r�r�r�r�r�)r)r<r�r�r�r�r�r$r$r%r�}s�zOwlv2VisionModel.forward�NNNFN)rCrDrErrHZmain_input_namerurr�r�rrr!rGr�rr@rr�r�r$r$r�r%rps0361������362�rcs�eZdZUeed<def�fdd�Ze�e	ddej	de363ej	dejfdd	���Ze�e	364ddej	de
dejfd
d���Ze							365		dde366ejde367ejde368ej	de369e
de370e
de371e
de
de372e
de373e
deeeffdd��Z�ZS)r�rmcs�t��|�t|jt�stdt|j��d���t|jt�s(tdt|j��d���|j}|j}|j	|_	|j374|_|j375|_t
|�|_t|�|_tj|j|j	dd�|_tj|j|j	dd�|_t�t�|j��|_|��dS)NzLconfig.text_config is expected to be of type Owlv2TextConfig but is of type �.zPconfig.vision_config is expected to be of type Owlv2VisionConfig but is of type F)rn)rtrur��text_configr�	TypeError�type�
vision_configrZprojection_dimrwr�r�r�r�rrrr�r�r�ryr!r�r�r�r�)r<rmrrr�r$r%ru�s0����376377zOwlv2Model.__init__Nr�r�rcC�|j||d�}|�|j�}|S)a�378        input_ids (`torch.LongTensor` of shape `(batch_size * num_max_text_queries, sequence_length)`):379            Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See380            [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input381            IDs?](../glossary#input-ids)382 383        Returns:384            text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by385            applying the projection layer to the pooled output of [`Owlv2TextModel`].386 387        Examples:388        ```python389        >>> import torch390        >>> from transformers import AutoProcessor, Owlv2Model391 392        >>> model = Owlv2Model.from_pretrained("google/owlv2-base-patch16-ensemble")393        >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16-ensemble")394        >>> inputs = processor(395        ...     text=[["a photo of a cat", "a photo of a dog"], ["photo of a astranaut"]], return_tensors="pt"396        ... )397        >>> with torch.inference_mode():398        ...     text_features = model.get_text_features(**inputs)399        ```)r�r�)r�r�r�)r<r�r��text_outputsZ
text_featuresr$r$r%�get_text_features�szOwlv2Model.get_text_featuresFr�r�cCr)av400        Returns:401            image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by402            applying the projection layer to the pooled output of [`Owlv2VisionModel`].403 404        Examples:405        ```python406        >>> import torch407        >>> from transformers.image_utils import load_image408        >>> from transformers import AutoProcessor, Owlv2Model409 410        >>> model = Owlv2Model.from_pretrained("google/owlv2-base-patch16-ensemble")411        >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16-ensemble")412 413        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"414        >>> image = load_image(url)415 416        >>> inputs = processor(images=image, return_tensors="pt")417        >>> with torch.inference_mode():418        ...     image_features = model.get_image_features(**inputs)419        ```�r�r�)rr�r�)r<r�r��vision_outputs�image_featuresr$r$r%�get_image_features�s�zOwlv2Model.get_image_features�return_lossr�r��return_base_image_embedsr�c420	Cs:|dur|n|jj}|dur|n|jj}|	dur|	n|jj}	|j|||||	d�}421|j|||||	d�}|d}|�|�}|422d}
|�|
�}
|
tj	j423|
dddd�}
|tj	j424|dddd�}|j���
|
j�}t�||
���|}|��}d}|r{t|�}|}|	s�||||
||425f}|dur�|f|S|St|||||
||426d	�S)427a4428        return_loss (`bool`, *optional*):429            Whether or not to return the contrastive loss.430        return_base_image_embeds (`bool`, *optional*):431            Whether or not to return the base image embeddings.432 433        Examples:434        ```python435        >>> from PIL import Image436        >>> import requests437        >>> from transformers import AutoProcessor, Owlv2Model438 439        >>> model = Owlv2Model.from_pretrained("google/owlv2-base-patch16-ensemble")440        >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16-ensemble")441        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"442        >>> image = Image.open(requests.get(url, stream=True).raw)443        >>> inputs = processor(text=[["a photo of a cat", "a photo of a dog"]], images=image, return_tensors="pt")444        >>> outputs = model(**inputs)445        >>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score446        >>> probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities447        ```NrrrrPrqT)�ordr��keepdim)r+r,r-r.r/r0r1)rmr�r�r�rr�r�r�r!�linalg�normr��expr�r�matmulr(r)r*)r<r�r�r�rr�r�r�rr�rrr.r/Ztext_embeds_normr�r-r,r+�outputr$r$r%r�sV#��	�448449�zOwlv2Model.forwardrIr�)	NNNNNNFNN)rCrDrErrHrurrr!rrrGrr�rr�rr@r*r�r�r$r$r�r%r��sl450 ����"����"��������	�451�452�r�cs>eZdZd453dedef�fdd�
Zdejdejfdd	�Z	�Z454S)�Owlv2BoxPredictionHead�rm�out_dimcsJt���|jj}t�||�|_t�||�|_t��|_	t�||�|_455dSrI)rtrurrwrr��dense0�dense1ZGELU�gelu�dense2)r<rmr!r�r�r$r%runs456457zOwlv2BoxPredictionHead.__init__rrcCs6|�|�}|�|�}|�|�}|�|�}|�|�}|SrI)r"r$r#r%)r<rrr$r$r%r�ws458459460461462zOwlv2BoxPredictionHead.forward)r )rCrDrErrMrur!rrGr�r�r$r$r�r%rms	rc	sPeZdZdef�fdd�Zdejdeejdeejde	ejfdd	�Z463�ZS)464�Owlv2ClassPredictionHeadrmcsZt���|jj}|jj|_t�|j|�|_t�|jd�|_	t�|jd�|_465t��|_dS)Nr)
rtrurrwr�	query_dimrr�r"�logit_shiftr�ZELU�elu)r<rmr!r�r$r%ru�s466467z!Owlv2ClassPredictionHead.__init__r/�query_embeds�468query_maskrcCs469|�|�}|dur%|j}|jdd�\}}t�|||jf��|�}||fS|tjj|ddd�d}|tjj|ddd�d}t�	d||�}|�470|�}	|�|�}471|�|472�d}473||	|474}|dur�|j
dkrmtj|dd	�}t�|d475kt�|j�j|�}|�tj�}||fS)NrPrqT)r�rg�����ư>z...pd,...qd->...pqrr�r�r)r"rr�r!Zzerosr'r�rr�einsumr(r�r)�ndimr��whereZfinforJrSrK)r<r/r*r+�image_class_embedsrr�r}�pred_logitsr(r�r$r$r%r��s&476477478479z Owlv2ClassPredictionHead.forward)rCrDrErrur!rGrrr@r�r�r$r$r�r%r&�s����r&cseZdZUeed<def�fdd�Zedededej	fdd��Z480d	ejdejfd481d�Ze
dd
�	d.dededeejdej	fdd��Z	d/dejdejdedejfdd�Z		d0dejdeejdeej	deejfdd�Z			d1dej	dejdej	deedeededeejfd d!�Z			d1dejdeedeededeejf482d"d#�Z	d/d$ejd%ejdedejfd&d'�Ze					d2dejd(eejdeedeeded)eedefd*d+��Ze					d2dej	dejdeej	deedeeded)eedefd,d-��Z�ZS)3�Owlv2ForObjectDetectionrmcs�t��|�t|�|_t|�|_t|�|_t|dd�|_t	j483|jj|jj
d�|_t	��|_||_|jjj|jjj|_|jjj|jjj|_|�|j|j�|_|��dS)Nr)r!r�)rtrur�r�r&�484class_headr�box_head�objectness_headrr�rrwr��485layer_normZSigmoid�sigmoidrmr|rv�num_patches_height�num_patches_width�compute_box_bias�box_biasr�r�r�r$r%ru�s486487488489z Owlv2ForObjectDetection.__init__r7r8rcCs�tjd|dtjd�}tjd|dtjd�}tj||dd�\}}tj||fdd�}|d|<|d|<|�dd	�}|S)490Nr)rJZxy)Zindexingrqr��.r�.rrP)r!r"rKZmeshgrid�stackr�)r7r8Z
x_coordinatesZ
y_coordinates�xx�yy�box_coordinatesr$r$r%�!normalize_grid_corner_coordinates�sz9Owlv2ForObjectDetection.normalize_grid_corner_coordinatesrcCs|��}|�|�}|d}|S)a#Predicts the probability that each image feature token is an object.491 492        Args:493            image_features (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_dim)`)):494                Features extracted from the image.495        Returns:496            Objectness scores.497        r;)�detachr4)r<rrdr$r$r%�objectness_predictor�s	498z,Owlv2ForObjectDetection.objectness_predictorrP)�maxsizeN�feature_mapc	Cs�|durtd��|�||�}t�|dd�}t�|d�t�|d�}t�|d�}|d|<|d|<t�|d�t�|d�}tj||gdd�}|S)	NzOfeature_map has been deprecated as an input. Please pass in num_patches insteadr�r�g-C��6?r;r<rqr�)r^rAr!Zclip�log�log1pZ	full_liker�)	r<r7r8rEr@Zbox_coord_biasZbox_sizeZ
box_size_biasr:r$r$r%r9�sz(Owlv2ForObjectDetection.compute_box_biasF�image_featsr�c	CsR|�|�}|r|j\}}}}|�||�}n|j}|�|j�}||7}|�|�}|S)a499        Args:500            image_feats:501                Features extracted from the image, returned by the `image_text_embedder` method.502            feature_map:503                A spatial re-arrangement of image_features, also returned by the `image_text_embedder` method.504            interpolate_pos_encoding:505                Whether to interpolate the pre-trained position encodings.506        Returns:507            pred_boxes:508                List of predicted boxes (cxcywh normalized to 0, 1) nested within a dictionary.509        )r3r�r9r:r�rr6)	r<rHrEr�rer�r7r8r:r$r$r%�
box_predictor�s510511z%Owlv2ForObjectDetection.box_predictorr*r+cCs|�|||�\}}||fS)a8512        Args:513            image_feats:514                Features extracted from the `image_text_embedder`.515            query_embeds:516                Text query embeddings.517            query_mask:518                Must be provided with query_embeddings. A mask indicating which query embeddings are valid.519        )r2)r<rHr*r+r0r/r$r$r%�class_predictor!sz'Owlv2ForObjectDetection.class_predictorr�r�r�r�r�c	Cs�|j||||||dd�}|r$|j\}}}	}520|	|jjj}|521|jjj}n|j}|j}|jd}
|jj�	|
�}t522�|dd�dd�dd�f|dd�dd�fj�}|dd�dd�dd�f|}|�|�}|jd|||jdf}|�
|�}|d}|||fS)NT)r�r�r�r�r�r�r�rrrq�����)r�r�rmrrvr7r8r1rrr!�broadcast_tor5r�)r<r�r�r�r�r�r�r�r�r�r�r7r8r�r/�class_token_out�new_sizer.r$r$r%�image_text_embedder5s8523�5245254526�527528z+Owlv2ForObjectDetection.image_text_embeddercCs�|jj||dd�}|r!|j\}}}}||jjj}	||jjj}529n|j}	|j}530|d}|jj�|�}t	�531|dd�dd�dd�f|dd�dd�fj�}
|dd�dd�dd�f|
}|�|�}|jd|	|532|jdf}|�|�}||fS)NT)r�r�r�rrrq)
r�rr�rmrrvr7r8rr!rLr5r�)r<r�r�r�r�rr�r�r�r7r8r�r/rMrNr$r$r%�image_embedderis*�4533�534z&Owlv2ForObjectDetection.image_embedder�query_image_features�query_feature_mapcCs:|�|�\}}|�|||�}t|�}g}g}	|j}535t|jd�D]f}tjgd�g|536d�}||}
t||
�\}}t�	|ddk�rEt537||
�}t�|�d}|d|k��}|�
�r�|||�d�}tj||dd�}t�d||�}|t�|�}|�|||�|	�|�q |r�t�|�}t�|	�}nd	\}}|||fS)538Nr)rrrrrr�g�������?r)Zaxiszd,id->i�NN)rJrIrrr�r�r!r�r\r]rarTZnonzeroZnumelZsqueezer�r,Zargmin�appendr=)r<rQrRr�r�rfreZpred_boxes_as_cornersZbest_class_embeds�best_box_indicesZpred_boxes_device�iZeach_query_boxZeach_query_pred_boxesZiousZ
iou_thresholdZ
selected_indsZselected_embeddingsZmean_embedsZmean_simZbest_box_indr*Zbox_indicesr$r$r%�embed_image_query�s6539540�541542z)Owlv2ForObjectDetection.embed_image_query�query_pixel_valuesr�c543Cs(|dur|n|jj}|dur|n|jj}|dur|n|jj}|j||d�d}|j||||d�\}}	|j\}544}}}
t�||545|||
f�}|j\}546}}}
t�||547|||
f�}|�|||�\}}}|j	||d�\}}|�548|||�}|s�|||||||	��f}tdd�|D��}|St
||||||d|	d�S)	a549        query_pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`):550            Pixel values of query image(s) to be detected. Pass in one query image per target image.551 552        Examples:553        ```python554        >>> import requests555        >>> from PIL import Image556        >>> import torch557        >>> from transformers import AutoProcessor, Owlv2ForObjectDetection558 559        >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16-ensemble")560        >>> model = Owlv2ForObjectDetection.from_pretrained("google/owlv2-base-patch16-ensemble")561 562        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"563        >>> image = Image.open(requests.get(url, stream=True).raw)564        >>> query_url = "http://images.cocodataset.org/val2017/000000001675.jpg"565        >>> query_image = Image.open(requests.get(query_url, stream=True).raw)566        >>> inputs = processor(images=image, query_images=query_image, return_tensors="pt")567 568        >>> # forward pass569        >>> with torch.no_grad():570        ...     outputs = model.image_guided_detection(**inputs)571 572        >>> target_sizes = torch.Tensor([image.size[::-1]])573 574        >>> # Convert outputs (bounding boxes and class logits) to Pascal VOC format (xmin, ymin, xmax, ymax)575        >>> results = processor.post_process_image_guided_detection(576        ...     outputs=outputs, threshold=0.9, nms_threshold=0.3, target_sizes=target_sizes577        ... )578        >>> i = 0  # Retrieve predictions for the first image579        >>> boxes, scores = results[i]["boxes"], results[i]["scores"]580        >>> for box, score in zip(boxes, scores):581        ...     box = [round(i, 2) for i in box.tolist()]582        ...     print(f"Detected similar object with confidence {round(score.item(), 3)} at location {box}")583        Detected similar object with confidence 0.938 at location [327.31, 54.94, 547.39, 268.06]584        Detected similar object with confidence 0.959 at location [5.78, 360.65, 619.12, 366.39]585        Detected similar object with confidence 0.902 at location [2.85, 360.01, 627.63, 380.8]586        Detected similar object with confidence 0.985 at location [176.98, -29.45, 672.69, 182.83]587        Detected similar object with confidence 1.0 at location [6.53, 14.35, 624.87, 470.82]588        Detected similar object with confidence 0.998 at location [579.98, 29.14, 615.49, 489.05]589        Detected similar object with confidence 0.985 at location [206.15, 10.53, 247.74, 466.01]590        Detected similar object with confidence 0.947 at location [18.62, 429.72, 646.5, 457.72]591        Detected similar object with confidence 0.996 at location [523.88, 20.69, 586.84, 483.18]592        Detected similar object with confidence 0.998 at location [3.39, 360.59, 617.29, 499.21]593        Detected similar object with confidence 0.969 at location [4.47, 449.05, 614.5, 474.76]594        Detected similar object with confidence 0.966 at location [31.44, 463.65, 654.66, 471.07]595        Detected similar object with confidence 0.924 at location [30.93, 468.07, 635.35, 475.39]596        ```Nrr)r�r�r�r�)rHr*csr�rIr$�r9�xr$r$r%r=(r�zAOwlv2ForObjectDetection.image_guided_detection.<locals>.<genexpr>)r/rirjrkrrfr0r1)rmr�r�r�rPr�r!r�rWrJrIr7r@rh)r<r�rXr�r�r�r�rRrErr�r7r8�597hidden_dimrHZquery_image_featsr*rUrkr0rfrjrr$r$r%�image_guided_detection�s^;���598��599��	�z.Owlv2ForObjectDetection.image_guided_detectionc600CsB|dur|n|jj}|dur|n|jj}|dur|n|jj}|j||||||d�\}}	}601|602j}|603j}|	j\}
}}}t�	|	|
|||f�}|jd|
}|�	|
||jd�}|�	|
||jd�}|ddk}|�604|||�\}}|�|�}|�||	|�}|s�|||||	||�
�|�
�f}tdd�|D��}|St|	|||||||d�S)	a�	605        input_ids (`torch.LongTensor` of shape `(batch_size * num_max_text_queries, sequence_length)`, *optional*):606            Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See607            [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input608            IDs?](../glossary#input-ids).609        output_hidden_states (`bool`, *optional*):610            Whether or not to return the last hidden state. See `text_model_last_hidden_state` and611            `vision_model_last_hidden_state` under returned tensors for more detail.612 613        Examples:614        ```python615        >>> import requests616        >>> from PIL import Image617        >>> import torch618 619        >>> from transformers import Owlv2Processor, Owlv2ForObjectDetection620 621        >>> processor = Owlv2Processor.from_pretrained("google/owlv2-base-patch16-ensemble")622        >>> model = Owlv2ForObjectDetection.from_pretrained("google/owlv2-base-patch16-ensemble")623 624        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"625        >>> image = Image.open(requests.get(url, stream=True).raw)626        >>> text_labels = [["a photo of a cat", "a photo of a dog"]]627        >>> inputs = processor(text=text_labels, images=image, return_tensors="pt")628        >>> outputs = model(**inputs)629 630        >>> # Target image sizes (height, width) to rescale box predictions [batch_size, 2]631        >>> target_sizes = torch.tensor([(image.height, image.width)])632        >>> # Convert outputs (bounding boxes and class logits) to Pascal VOC format (xmin, ymin, xmax, ymax)633        >>> results = processor.post_process_grounded_object_detection(634        ...     outputs=outputs, target_sizes=target_sizes, threshold=0.1, text_labels=text_labels635        ... )636        >>> # Retrieve predictions for the first image for the corresponding text queries637        >>> result = results[0]638        >>> boxes, scores, text_labels = result["boxes"], result["scores"], result["text_labels"]639        >>> for box, score, text_label in zip(boxes, scores, text_labels):640        ...     box = [round(i, 2) for i in box.tolist()]641        ...     print(f"Detected {text_label} with confidence {round(score.item(), 3)} at location {box}")642        Detected a photo of a cat with confidence 0.614 at location [341.67, 23.39, 642.32, 371.35]643        Detected a photo of a cat with confidence 0.665 at location [6.75, 51.96, 326.62, 473.13]644        ```N)r�r�r�r�r�r�rrqr;csr�rIr$rYr$r$r%r=�r�z2Owlv2ForObjectDetection.forward.<locals>.<genexpr>)r/r.rerrdrfr0r1)rmr�r�r�rOr0r1r�r!r�rJrCrIr7r@rb)r<r�r�r�r�r�r�r�r*rEr�rrr�r7r8r[rHZmax_text_queriesr+r0rfrdrerr$r$r%r�6sZ4��645646�647�zOwlv2ForObjectDetection.forwardrIr�rSr�r	)rCrDrErrHru�staticmethodrMr!rrArGrCrrr9r�rIr@rJrOrPrWrrhr\rbr�r�r$r$r�r%r1�s�648���������649�%����650��������651�7�����652�/����653�,��������u��������	�r1)r�r�r�rr1)BrF�dataclassesr�	functoolsr�typingrrrr!rrZactivationsr654Zmodeling_attn_mask_utilsrrZmodeling_layersr
Zmodeling_outputsrrZmodeling_utilsr�utilsrrrrrrZconfiguration_owlv2rrrZtransformers.image_transformsrZ655get_loggerrC�loggerr&r)r*rNrQr\rarbrhr�rlr�r�r�r�r�r�r�r�rrr�rr&r1�__all__r$r$r$r%�<module>sr 656%	�2�.N l20QM781K0~