Aluode/PerceptionLabPortable
0
1o
2 .�Yi� � @ s d dl mZmZ d dlmZ d dlmZmZ d dlZddl m3Z4 ddl m Z m
Z
mZmZmZ e � r7d dlmZ ed d5d �Ze�e�ZG d d
� d
e�ZG dd� de�ZG dd� de�ZG dd� de�ZG dd� de�Z G dd� de�Z G dd� de �Z G dd� de �Z G d d � d �Z G d d � d e �Z!G d d!� d!e �Z"G d"d#� d#e �Z#G d$d%� d%e �Z$G d&d'� d'e �Z%G d(d)� d)e �Z&G d*d+� d+e!�Z'G d,d-� d-e"�Z(G d.d/� d/e"�Z)G d0d1� d1e"�Z*G d2d3� d3e"�Z+G d4d5� d5e"�Z,G d6d7� d7e#�Z-G d8d9� d9e#�Z.G d:d;� d;e �Z/dS )<� )�ABC�abstractmethod)�Iterable)�Any�OptionalN� )�PretrainedConfig)�is_hqq_available�is_quanto_greater�is_torch_greater_or_equal�is_torchdynamo_compiling�logging)� Quantizerz2.7T�Z6accept_devc @ s� e Zd ZdZdZdd� Zdd� Zedej fdd ��Z7e 8d dej d ej d e e e
ef d
eej ej f fdd��Zedej d
eeef fdd��Zed
efdd��Zed
efdd��Zdd� Zdd� Zd!dd �Zd ejd
d9fd d �Zd10S )"�CacheLayerMixinz0Base, abstract class for a single layer's cache.Fc C s d | _ d | _d| _d S �NF)�keys�values�is_initialized��self� r ��E:\DocsHouse\542 percep lab latest\PerceptionLab\PerceptionLab_Portable\python_embed\Lib\site-packages\transformers/cache_utils.py�__init__ s 11zCacheLayerMixin.__init__c C s12 | j j� S �N)� __class__�__name__r r r r �__repr__$ s 13zCacheLayerMixin.__repr__�14key_statesc C � d S r r �r r r r r �lazy_initialization' � z#CacheLayerMixin.lazy_initializationN� value_states� cache_kwargs�returnc C r r r �r r r# r$ r r r �update* s zCacheLayerMixin.update�cache_positionc C r r r )r r( r r r �get_mask_sizes/ r" z CacheLayerMixin.get_mask_sizesc C r r r r r r r �get_seq_length2 r" z CacheLayerMixin.get_seq_lengthc C r r r r r r r �get_max_cache_shape5 r" z#CacheLayerMixin.get_max_cache_shapec C s2 | j r| jjddd�| _| jjddd�| _dS dS )z(Offload this layer's data to CPU device.�cpuT�Z non_blockingN)r r �tor r r r r �offload8 s �zCacheLayerMixin.offloadc C sH | j r | jj| jkr"| jj| jdd�| _| jj| jdd�| _dS dS dS )zcIn case of layer offloading, this allows to move the data back to the layer's device ahead of time.Tr- N)r r �devicer. r r r r r �prefetch> s �zCacheLayerMixin.prefetchc C s2 | j r
| j�� | j�� t| d�rd| _dS dS )z4Resets the cache values while preserving the objects�cumulative_lengthr N)r r Zzero_r �hasattrr2 r r r r �resetD s 15161718�zCacheLayerMixin.reset�beam_idxc C sH | � � dkr"| j�d|�| jj��| _| j�d|�| jj��| _dS dS )z,Reorders this layer's cache for beam search.r N)r* r Z index_selectr. r0 r �r r5 r r r �
reorder_cacheM s �z CacheLayerMixin.reorder_cacher �r% N)r �19__module__� __qualname__�__doc__�is_compileabler r r �torch�Tensorr! r �dict�strr �tupler' �intr) r* r+ r/ r1 r4 �20LongTensorr7 r r r r r s8 ���� � 21 r c @ s� e Zd ZdZdZdejfdd�Z d dejdejdee e22e f d e ejejf fd23d �Z
d ejd e eef fd
d�Zd efdd�Zd efdd�Zded dfdd�Zded dfdd�Zdejd dfdd�ZdS ) � DynamicLayerz�24 A cache layer that grows dynamically as more tokens are generated. This is the default for generative models.25 It stores the key and value states as tensors of shape `[batch_size, num_heads, seq_len, head_dim]`.26 Fr c C sH |j |j| _ | _tjg | j | jd�| _tjg | j | jd�| _d| _d S )N��dtyper0 T)rF r0 r= �tensorr r r r r r r r! \ s 27z DynamicLayer.lazy_initializationNr# r$ r% c C sH | j s| �|� tj| j|gdd�| _tj| j|gdd�| _| j| jfS )�� 28 Update the key and value caches in-place, and return the necessary keys and value states.29 30 Args:31 key_states (`torch.Tensor`): The new key states to cache.32 value_states (`torch.Tensor`): The new value states to cache.33 cache_kwargs (`dict[str, Any]`, *optional*): Additional arguments for the cache.34 35 Returns:36 tuple[`torch.Tensor`, `torch.Tensor`]: The key and value states.37 �������dim)r r! r= �catr r r& r r r r' b s38 39 zDynamicLayer.updater( c C s" d}|j d }| �� | }||fS )zDReturn the length and offset of the cache, used to generate the maskr )�shaper* )r r( � kv_offset� query_length� kv_lengthr r r r) { s 40 zDynamicLayer.get_mask_sizesc C s$ | j r41| j�� dkr dS | jjd S )�1Returns the sequence length of the cached states.r rI )r r ZnumelrM r r r r r* � s zDynamicLayer.get_seq_lengthc C s dS )zeReturns the maximum sequence length of the cache object. DynamicLayer does not have a maximum length.�����r r r r r r+ � s z DynamicLayer.get_max_cache_shape�42max_lengthc C s` |dk r | � � t|� }| � � |krdS | jdd|�dd�f | _| jdd|�dd�f | _dS )z�43 Crop the past key values up to a new `max_length` in terms of tokens. `max_length` can also be negative44 to remove `max_length` tokens.45 r N.)r* �absr r �r rS r r r �crop� s zDynamicLayer.crop�repeatsc C s8 | � � dkr| jj|dd�| _| jj|dd�| _dS dS )z8Repeat the cache `repeats` times in the batch dimension.r rJ N)r* r Zrepeat_interleaver �r rW r r r �batch_repeat_interleave� s �z$DynamicLayer.batch_repeat_interleave�indicesc C s4 | � � dkr| j|df | _| j|df | _dS dS )z<Only keep the `indices` in the batch dimension of the cache.r .N)r* r r �r rZ r r r �batch_select_indices� s �z!DynamicLayer.batch_select_indicesr )r r9 r: r; �46is_slidingr= r> r! r r? r@ r rA r' rB r) r* r+ rV rY r\ r r r r rD T s( 47����48� rD c s� e Zd ZdZdZdef� fdd� Z ddejdejd e e49e e f d50e
ejejf fd d �Zd
ejd51e
eef fdd�Zd52efdd�Zd53efdd�Zded54df� fdd� Z� ZS )�DynamicSlidingWindowLayerz�55 A cache layer that grows dynamically as more tokens are generated, up until the sliding window size.56 It stores the key and value states as tensors of shape `[batch_size, num_heads, min(seq_len, sliding_window), head_dim]`.57 T�sliding_windowc s t � �� || _d| _d S �Nr )�superr r_ r2 )r r_ �r r r r � s 5859z"DynamicSlidingWindowLayer.__init__Nr r# r$ r% c C s� | j s| �|� | j|jd 7 _tj| j|gdd�}tj| j|gdd�}|dd�dd�| j d d�dd�f | _|dd�dd�| j d d�dd�f | _||fS )rH rI rJ Nr ) r r! r2 rM r= rL r r r_ )r r r# r$ �full_key_states�full_value_statesr r r r' � s 60**z DynamicSlidingWindowLayer.updater( c C sX |j d }| j| jk}t| j| j d d�}|r#| jd | }||fS | j| }||fS ��NReturn the length and offset of the cache, used to generate the attention maskr r )rM r2 r_ �max)r r( rO �is_fullrN rP r r r r) � s 61 62�z(DynamicSlidingWindowLayer.get_mask_sizesc C � | j S �rQ �r2 r r r r r* � � z(DynamicSlidingWindowLayer.get_seq_lengthc C ri �z+Return the maximum cache shape of the cache�r_ r r r r r+ � rl z-DynamicSlidingWindowLayer.get_max_cache_shaperS c s4 | � � | jkr td��t� �|� | jjd | _dS )z�63 Crop the past key values up to a new `max_length` in terms of tokens. `max_length` can also be64 negative to remove `max_length` tokens.65 z�Cannot `crop` a `DynamicSlidingWindowLayer` after it has seen more tokens than itssliding window (otherwise some states are lost)rI N)r* r_ �66ValueErrorra rV r rM r2 rU rb r r rV � s � z DynamicSlidingWindowLayer.cropr )r r9 r: r; r] rB r r= r> r r? r@ r rA r' r) r* r+ rV �
__classcell__r r rb r r^ � s$ ����67� !
r^ c s� e Zd ZdZdZdZdef� fdd� Zdej fdd �Z68 69ddej d ej d e e e
ef d
eej ej f fdd�Zdej d
eeef fdd�Zd
efdd�Zd
efdd�Z� ZS )� StaticLayera� 70 A static cache layer that stores the key and value states as static tensors of shape `[batch_size, num_heads, max_cache_len), head_dim]`.71 It lazily allocates its full backing tensors, and then mutates them in-place. Built for `torch.compile` support.72 73 Args:74 max_cache_len (`int`):75 Maximum number of tokens that can be stored, used for tensor preallocation.76 TF�
max_cache_lenc s t � �� || _d S r )ra r rr )r rr rb r r r s 7778zStaticLayer.__init__r c C s� |j \| _| _}| _|j|j| _| _tj| j| j| j| jf| j| jd�| _ tj| j| j| j| jf| j| jd�| _79t � sJtj �
| j � tj �
| j80� d| _dS )a6 81 Lazy initialization of the keys and values tensors. This allows to get all properties (dtype, device,82 num_heads in case of TP etc...) at runtime directly, which is extremely practical as it avoids moving83 devices, dtypes etc later on for each `update` (which could break the static dynamo addresses as well).84 85 If this is unwanted, one can call `early_initialization(...)` on the Cache directly, which will call this86 function ahead-of-time (this is required for `torch.export` for example). Note that for `compile`, as we87 internally don't compile the prefill, this is guaranteed to have been called already when compiling.88 If compiling the prefill as well, e.g. calling `model.compile(...)` before `generate` with a static cache,89 it is still supported in general, but without guarantees depending on the compilation options (e.g. cuda graphs,90 i.e. `mode="reduce-overhead"` is known to fail). But it will in general work correctly, and prefill should91 not be compiled anyway for performances!92 rE TN)rM �max_batch_size� num_heads�head_dimrF r0 r= �zerosrr r r r Z_dynamoZmark_static_addressr )r r �_r r r r! s �� 93z StaticLayer.lazy_initializationNr# r$ r% c C s� | j s| �|� |dur|�d�nd}|dur|n94tj|jd | jd�}z| j�d||� | j �d||� W n! t95yW || jdd�dd�|f< || j dd�dd�|f< Y nw | j| j fS )rH Nr( rI �r0 � ) r r! �getr= �arangerM r0 r � index_copy_r �NotImplementedError)r r r# r$ r( r r r r' . s 96 � � zStaticLayer.updater( c C s d}| j }||fS )rf r �rr )r r( rN rP r r r r) T s zStaticLayer.get_mask_sizesc C s | j r| jd jdd��� S dS )rQ )r r rR rJ r )r r �any�sumr r r r r* Z s zStaticLayer.get_seq_lengthc C ri rm r~ r r r r r+ ` rl z StaticLayer.get_max_cache_shaper )r r9 r: r; r< r] rB r r= r> r! r r? r@ r rA r' r) r* r+ rp r r rb r rq � s& )����97� &rq c s� e Zd ZdZdZdedef� fdd� Z ddejd ejd98e e99e e f d e
ejejf fd d
�Zdejd e
eef fdd�Zd efdd�Z� ZS )�StaticSlidingWindowLayera� 100 A static cache layer that stores the key and value states as static tensors of shape101 `[batch_size, num_heads, min(max_cache_len, sliding_window), head_dim]`. It lazily allocates its full backing102 tensors, and then mutates them in-place. Built for `torch.compile` support.103 104 Args:105 max_cache_len (`int`):106 Maximum number of tokens that can be stored, used for tensor preallocation.107 sliding_window (`int`):108 The size of the sliding window.109 Trr r_ c s" t ||�}t� j|d� d| _d S )Nr~ r )�minra r r2 )r rr r_ Zeffective_max_cache_lenrb r r r t s 110111z!StaticSlidingWindowLayer.__init__Nr r# r$ r% c C s� | j s| �|� |dur|�d�nd}|dur|n112tj|jd | jd�}| j}|| jk}| j|jd 7 _|r�|jd dkr�| j j113ddd�}| j j114ddd�}tj dgt
| jd�} ||dd�dd�| f< ||dd�dd�| f< | j �|� | j �|� | j | j fS tj| j dd�dd�dd�dd�f |fdd �}115tj| j dd�dd�dd�dd�f |fdd �} n||jd116 | jkr�|d kr�|}117|} nltj| j dd�dd�d|�dd�f |fdd �}118tj| j dd�dd�d|�dd�f |fdd �} n;z| j �d119||� | j �d120||� W n" t�y* || j dd�dd�|f< || j dd�dd�|f< Y nw | j | j fS | j �|121dd�dd�| j d�dd�f � | j �| dd�dd�| j d�dd�f � |122| fS ) rH Nr( rI rx r rR )�dimsrE rJ ry r )r r! rz r= r{ rM r0 r2 rr r Zrollr rG rB Zcopy_rL r| r} ) r r r# r$ r( r2 rh Znew_keys�123new_values�indexrc rd r r r r' y sJ 124 �125 0202 � ,,z StaticSlidingWindowLayer.updater( c C st |j d }| j}| j| jk}t| j| d d�}|r$|| d }||fS | j| |kr4| j| }||fS |}||fS re )rM rr r2 rg )r r( rO r_ rh rN rP r r r r) � s 126 �127�z'StaticSlidingWindowLayer.get_mask_sizesc C ri rj rk r r r r r* � rl z'StaticSlidingWindowLayer.get_seq_lengthr )r r9 r: r; r] rB r r= r> r r? r@ r rA r' r) r* rp r r rb r r� e s ����128� Qr� c s� e Zd ZdZ ddededed ed129ef130� fd d �
Z
ddejdejdee e131e f de ejejf fdd�Z
edd� �Zedd� �Zdefdd�Z� ZS ) �QuantizedLayera 132 A quantized layer similar to what is described in the [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache paper](https://huggingface.co/papers/2402.02750).133 It allows the model to generate longer sequence length without allocating too much memory for the key and value caches by134 applying quantization.135 136 The cache has two types of storage, one for original precision and one for the quantized cache. A `residual length`137 is set as a maximum capacity for the original precision cache. When the length goes beyond maximum capacity, the original138 precision cache is discarded and moved into the quantized cache. The quantization is done per-channel with a set `q_group_size`139 for both Keys and Values, in contrast to what was described in the paper.140 � r �@ � �nbits�axis_key�141axis_value� q_group_size�residual_lengthc s2 t � �� || _|| _|| _|| _|| _d| _d S r` )ra r r� r� r� r� r� r2 �r r� r� r� r� r� rb r r r � s 142143zQuantizedLayer.__init__Nr r# r$ r% c C sT | j |jd 7 _ | js,| �|� | j|�� | jd�| _| j|�� | jd�| _ ||fS | �144| j�}| �145| j �}t j || j
|gdd�}t j || j|gdd�}| j
�� dkr�| j
jd d | jkr�| j|�� | jd�| _| j|�� | jd�| _ t jg |j|jd�| _
t jg |j|jd�| _||fS t j | j
|gdd�| _
t j | j|gdd�| _||fS )rH rI )�axisrJ r� r rE )r2 rM r r! � _quantize�146contiguousr� Z_quantized_keysr� Z_quantized_values� _dequantizer= rL r r rK r� rG rF r0 )r r r# r$ Z dequant_keysZdequant_valuesZkeys_to_returnZvalues_to_returnr r r r' � s& 147 $�zQuantizedLayer.updatec C r r r )r rG r� r r r r� ' r" zQuantizedLayer._quantizec C r r r )r Zq_tensorr r r r� * r" zQuantizedLayer._dequantizec C ri rj rk r r r r r* - rl z QuantizedLayer.get_seq_length�r� r r r� r� r )r r9 r: r; rB r r= r> r r? r@ r rA r' r r� r� r* rp r r rb r r� � sB
����������148�)149150r� c �N e Zd Z ddedededed ef151� fd152d �
Zd d
� Zdd� Z� ZS )�QuantoQuantizedLayerr� r r� r� r� r� r� r� r� c s� t � j|||||d� tddd�r ddlm}m}m} ntd��| jdvr-t d | j� ���| j153d154vr:t d | j155� ���| j d156vrGt d | j � ���| jd
krN|n|| _ |� | _
d S )N�r� r� r� r� r� z0.2.5Tr r )� MaxOptimizer�qint2�qint4ziYou need optimum-quanto package version to be greater or equal than 0.2.5 to use `QuantoQuantizedCache`. )ry r� zA`nbits` for `quanto` backend has to be one of [`2`, `4`] but got )r rR zE`axis_key` for `quanto` backend has to be one of [`0`, `-1`] but got zG`axis_value` for `quanto` backend has to be one of [`0`, `-1`] but got r� )ra r r157 �optimum.quantor� r� r� � ImportErrorr� ro r� r� �qtype� optimizer) r r� r� r� r� r� r� r� r� rb r r r 3 s, � �158159160161� z QuantoQuantizedLayer.__init__c C s> ddl m} | �|| j|| j�\}}||| j|||| j�}|S )Nr )�quantize_weight)r� r� r� r� r� )r rG r� r� �scaleZ zeropoint�qtensorr r r r� Y s z QuantoQuantizedLayer._quantizec C s |� � S r )�162dequantize)r r� r r r r� ` s z QuantoQuantizedLayer._dequantizer� �r r9 r: rB r r� r� rp r r rb r r� 2 s& ������&r� c r� )�HQQQuantizedLayerr� r r� r� r� r� r� r� r� c s| t � j|||||d� t� std��| jdvr td| j� ���| jdvr,td| j� ���| jdvr9td| j� ���t| _ d S )Nr� z4You need to install `hqq` to use `HQQQuantizedLayer`)r ry � r� � zM`nbits` for `HQQ` backend has to be one of [`1`, `2`, `3`, `4`, `8`] but got )r r zA`axis_key` for `HQQ` backend has to be one of [`0`, `1`] but got zC`axis_value` for `HQQ` backend has to be one of [`0`, `1`] but got )163ra r r r� r� ro r� r� � HQQQuantizer� quantizerr� rb r r r e s$ �164165�166167168zHQQQuantizedLayer.__init__c C sz | j j||| jj| jj| j| jd�\}}| jj|d< | j j||| jjd� |d �|j�|d< |d �|j�|d< ||fS )N)r� r0 �
compute_dtyper� Z169group_sizer� )�metar0 r� �zero) r� �quantizer r0 rF r� r� �cudar. )r rG r� r� r� r r r r� � s 170� zHQQQuantizedLayer._quantizec C s |\}}| j �||�}|S r )r� r� )r r� Z quant_tensorr� rG r r r r� � s z HQQQuantizedLayer._dequantizer� r� r r rb r r� d s& ������ r� c
@ s� e Zd ZdZ dFdeee deee dedefd d171�Z d d � Z172dGd
e defdd�Z dGd
e defdd�Z
dHdejdejd
e deeeef deejejf f173dd�Zde de de d ejd ejf174d d �ZdId
e de fd!d"�Zd#ejd
e dee e f fd$d%�ZdId
e de fd&d'�Zd(d)� Zd*ej fd+d,�Z d-e fd.d/�Z d0e fd1d2�Z d3ejfd4d5�Z e!de fd6d7��Z"e!de fd8d9��Z#e!defd:d;��Z$e!defd<d=��Z%e!dee fd>d?��Z&d
e deejejf fd@dA�Z'dBdC� Z(dDdE� Z)dS )J�Cachean 175 A `Cache` is mostly a list of `CacheLayerMixin` objects, one per model layer. It serves as a container for176 the Cache of each layer.177 178 Args:179 layers (`Optional`, *optional*):180 A list of pre-created `CacheLayerMixin`. If omitted (`None`), then `layer_class_to_replicate` will181 be used.182 layer_class_to_replicate (`type[CacheLayerMixin]`, *optional*):183 Only used if `layers` is omitted (`None`), in which case it will be used as the base class for each layer,184 and the layers will be added lazily as soon as `update` is called with a `layer_idx` greater than the current185 list of layers.186 offloading (`bool`, *optional*, defaults to `False`):187 Whether to perform offloading of the layers to `cpu`, to save GPU memory.188 offload_only_non_sliding (`bool`, *optional*, defaults to `True`):189 If `offloading` is `True`, this further decides if only the non-sliding layers will be offloaded (because190 usually the sliding layers are small in size, so there is no need to offload them, and skipping it is faster).191 NFT�layers�layer_class_to_replicate�192offloading�offload_only_non_slidingc C sz |d ur |d ur t d��|d u r|d u rt d��|d ur |ng | _|| _|| _| jr;|| _tr3t�� ntj�� | _ d S d S )Na You can construct a Cache either from a list `layers` of all the predefined `CacheLayer`, or from a `layer_class_to_replicate`, in which case the Cache will append a new layer corresponding to `layer_class_to_replicate` for each new call to `update` with an idx not already in the Cache.z_You should provide exactly one of `layers` or `layer_class_to_replicate` to initialize a Cache.)193ro r� r� r� �only_non_sliding�#_is_torch_greater_or_equal_than_2_7r= ZStreamr� �prefetch_stream)r r� r� r� r� r r r r � s �� �zCache.__init__c C s | j j� d| j� d�S )Nz(layers=�))r r r� r r r r r � s zCache.__repr__� layer_idxr� c C s� |r z|| j |d� �d� }W n ty | j �d�}Y n w |t| j�k r)|nd}tr0| jntj� | j�� | j| �194� W d � dS 1 sJw Y dS )a< 195 Prefetch a given layer on its device. If `only_non_sliding` is True, it will try to prefetch only the layers196 which are non-sliding. If the `layer_idx` is outside the range, this will circle back to the first layers.197 Note that we use a non-default stream for this, to avoid blocking.198 NFr ) r] r� ro �lenr� r� r� r= r� �streamr1 �r r� r� r r r r1 � s �"�zCache.prefetchc C s$ |r| j | s| j| �� dS dS )a 199 Offload a given `layer_idx`. If `only_non_sliding` is True, it will offload `layer_idx` only if it is a200 non-sliding layer. Note that we do it on the default stream, so that we ensure all earlier201 computation in the layer's `update` methods are finished.202 N)r] r� r/ r� r r r r/ � s �z
Cache.offloadr r# r$ r% c C s� | j durt| j�|kr| j�| � � � t| j�|ks | jr2tj�|j�� | j203� | � |d | j � | j| �
|||�\}}| jrH| �|| j � ||fS )a� 204 Updates the cache with the new `key_states` and `value_states` for the layer `layer_idx`.205 206 Parameters:207 key_states (`torch.Tensor`):208 The new key states to cache.209 value_states (`torch.Tensor`):210 The new value states to cache.211 layer_idx (`int`):212 The index of the layer to cache the states for.213 cache_kwargs (`dict[str, Any]`, *optional*):214 Additional arguments for the cache subclass. These are specific to each subclass and allow new types of215 cache to be created.216 217 Return:218 A tuple containing the updated key and value states.219 Nr )r� r� r� �appendr� r= r� Zdefault_streamr0 Z wait_streamr� r1 r� r' r/ )r r r# r� r$ r r r r r r' � s 220�z Cache.update�221batch_sizert ru rF r0 c C s2 t j||d|f||d�}| jD ]}|�|� qdS )z�222 Initialize all the layers in advance (it's otherwise lazily initialized on the first `update` call).223 This is useful for our `export` recipes, as `export` needs everything in advance.224 r rE N)r= rv r� r! )r r� rt ru rF r0 Zfake_keys_tensor�layerr r r �early_initialization s 225 226 �zCache.early_initializationr c C � |t | j�kr dS | j| �� S )z=Returns the sequence length of the cache for the given layer.r )r� r� r* �r r� r r r r* s zCache.get_seq_lengthr( c C s, |t | j�kr|jd dfS | j| �|�S )a 227 Return a tuple (kv_length, kv_offset) corresponding to the length and offset that will be returned for228 the given layer at `layer_idx`.229 The masks are then prepared according to the given lengths (kv_length, kv_offset) and patterns for each layer.230 r )r� r� rM r) �r r( r� r r r r) $ s zCache.get_mask_sizesc C r� )zaReturns maximum sequence length of the cache object. Dynamic caches do not have a maximum length.rR )r� r� r+ r� r r r r+ 0 s zCache.get_max_cache_shapec C s& t t| j��D ] }| j| �� qdS )z$Recursively reset all layers tensorsN)�ranger� r� r4 r� r r r r4 8 s �z Cache.resetr5 c C �( t t| j��D ]231}| j| �|� qdS )z!Reorder the cache for beam searchN)r� r� r� r7 )r r5 r� r r r r7 = � �zCache.reorder_cacherS c C r� )z"Crop the cache to the given lengthN)r� r� r� rV )r rS r� r r r rV B r� z232Cache.croprW c C r� )z Repeat and interleave the cacheN)r� r� r� rY )r rW r� r r r rY G r� z Cache.batch_repeat_interleaverZ c C r� )z Select indices from the cacheN)r� r� r� r\ )r rZ r� r r r r\ L r� zCache.batch_select_indicesc C s6 dd� | j D �}tt|��dkrtd|� ���|d S )z*Return the maximum batch size of the cachec S � g | ]}|j �qS r )rs ��.0r� r r r �233<listcomp>T � z(Cache.max_batch_size.<locals>.<listcomp>r z0Max batch size is not consistent across layers: r )r� r� �setro �r r r r r rs Q s zCache.max_batch_sizec C s dd� | j D �}t|�S )z,Return the maximum cache length of the cachec S r� r r~ r� r r r r� \ r� z'Cache.max_cache_len.<locals>.<listcomp>)r� rg r� r r r rr Y s zCache.max_cache_lenc C s& t | j�dkr dS tdd� | jD ��S )z'Return whether the cache is compileabler Fc s � � | ]}|j V qd S r )r< r� r r r � <genexpr>e � � z'Cache.is_compileable.<locals>.<genexpr>�r� r� �allr r r r r< _ s zCache.is_compileablec C s" t | j�dkotdd� | jD ��S )z,Return whether the cache data is initializedr c s r� r )r r� r r r r� j r� z'Cache.is_initialized.<locals>.<genexpr>r� r r r r r g s "zCache.is_initializedc C s dd� | j D �S )z9Return whether the layers of the cache are sliding windowc S s g | ]}t |d d��qS )r] F)�getattrr� r r r r� o s z$Cache.is_sliding.<locals>.<listcomp>�r� r r r r r] l s zCache.is_slidingc C s@ |t | j�k r| j| j| j| jfS tdt | j�� d|� ����z�234 Support for backwards-compatible `past_key_values` indexing, e.g. `past_key_values[0][0].shape[2]` to get the235 sequence length.236 zCache only has z. layers, attempted to access layer with index )r� r� r r �KeyErrorr� r r r � __getitem__q s237 �zCache.__getitem__c c s2 � t t| ��D ]}| j| j| j| jfV qdS �z�238 Support for backwards-compatible `past_key_values` iteration, e.g. `for x in past_key_values:` to iterate over239 keys and values240 N)r� r� r� r r r� r r r �__iter__} s � �zCache.__iter__c C �241 t | j�S )zN242 This value corresponds to the number of layers in the model.243 )r� r� r r r r �__len__� s 244z
Cache.__len__)NNFT)Tr �r )*r r9 r: r; r �listr �type�boolr r rB r1 r/ r= r> r? r@ r rA r' rF r0 r� r* r) r+ r4 rC r7 rV rY r\ �propertyrs rr r< r r] r� r� r� r r r r r� � s| �245�246��247������248�)����249� r� c s� e Zd ZdZ ddeeeejejf dee de250de251f� fdd �
Z d252eeejejf fd d �Z e
d
eeejejf d253d fdd��Z� ZS )� DynamicCachea*254 255 A cache that grows dynamically as more tokens are generated. This is the default for generative models.256 It stores the key and value states as a list of `CacheLayer`, one for each layer. The expected shape for each tensor257 in the `CacheLayer`s is `[batch_size, num_heads, seq_len, head_dim]`.258 If a config is passed, it will additionally check for sliding or hybrid cache structure, greatly reducing the259 memory requirement of the cached tensors to `[batch_size, num_heads, min(seq_len, sliding_window), head_dim]`.260 261 See `Cache` for details on common methods that are implemented by all cache classes.262 263 Args:264 ddp_cache_data (`Iterable[tuple[torch.Tensor, torch.Tensor]]`, *optional*):265 It was originally added for compatibility with `torch.distributed` (DDP). In a nutshell, it is266 `map(gather_map, zip(*caches))`, i.e. each item in the iterable contains the key and value states267 for a layer gathered across replicas by torch.distributed (shape=[global batch size, num_heads, seq_len, head_dim]).268 Note: it needs to be the 1st arg as well to work correctly269 config (`PretrainedConfig`, *optional*):270 The config of the model for which this Cache will be used. If passed, it will be used to check for sliding271 or hybrid layer structure, greatly reducing the memory requirement of the cached tensors to272 `[batch_size, num_heads, min(seq_len, sliding_window), head_dim]`.273 offloading (`bool`, *optional*, defaults to `False`):274 Whether to perform offloading of the layers to `cpu`, to save GPU memory.275 offload_only_non_sliding (`bool`, *optional*, defaults to `False`):276 If `offloading` is `True`, this further decides if only the non-sliding layers will be offloaded (because277 usually the sliding layers are small in size, so there is no need to offload them, and skipping it is faster).278 279 Example:280 281 ```python282 >>> from transformers import AutoTokenizer, AutoModelForCausalLM, DynamicCache283 284 >>> model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2-0.5B-Instruct")285 >>> tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-0.5B-Instruct")286 287 >>> inputs = tokenizer(text="My name is Qwen2", return_tensors="pt")288 289 >>> # Prepare a cache class and pass it to model's forward290 >>> past_key_values = DynamicCache(config=model.config)291 >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True)292 >>> outputs.past_key_values # access cache filled with key/values from generation293 ```294 NF�ddp_cache_data�configr� r� c
s$ g }|d urS|j dd�}t|dd �pt|dd �� t|dd �}|d u r.� fdd�t|j�D �}t|d�r;|d |j � }|D ]}|d v rL|�t� d295�� q=|�t� � q=|d urvt |�D ]\} \}296} |d u rk|�t� � || �297|298| �\} } q[t |�d kr�t � j
t||d � d S t � j
|||d
� d S )NT��decoderr_ �attention_chunk_size� layer_typesc s g | ]299}� d ur300dnd�qS )N�sliding_attention�full_attentionr �r� rw rn r r r� � � ��z)DynamicCache.__init__.<locals>.<listcomp>�num_kv_shared_layers)r� �chunked_attentionrn r )r� r� r� �r� r� r� )�get_text_configr� r� �num_hidden_layersr3 r� r� r^ rD � enumerater' r� ra r )
r r� r� r� r� r� Zdecoder_configr� �301layer_typer� r r# rw rb rn r r � s: � 302�303 304�zDynamicCache.__init__r% c C s& d}| j D ] }||j|jff7 }q|S )z�305 Converts the `Cache` instance into the its equivalent in the legacy cache format. Used for306 backward compatibility.307 r )r� r r )r � legacy_cacher� r r r �to_legacy_cache� s 308z DynamicCache.to_legacy_cache�past_key_valuesc C sP | � }|du r t �d� |dur&tt|��D ]}|| \}}|�|||� q|S )z�309 Converts a cache in the legacy cache format into an equivalent `Cache`. Used for310 backward compatibility.311 N�9past_key_values should not be None in from_legacy_cache())�logger� warning_oncer� r� r' )�clsr� �cacher� r r# r r r �from_legacy_cache� s 312 z DynamicCache.from_legacy_cache)NNFF)r r9 r: r; r r rA r= r> r r� r r� � classmethodr� rp r r rb r r� � s$ ,����� 4313,r� c s8 e Zd ZdZ d314dedededef� fdd �
Z� ZS ) � StaticCachea� 315 Static Cache class to be used with `torch.compile(model)` and `torch.export()`. It will check the `config`316 for potential hybrid cache structure, and initialize each layer accordingly.317 318 See `Cache` for details on common methods that are implemented by all cache classes.319 320 Args:321 config (`PretrainedConfig`):322 The config of the model for which this Cache will be used. It will be used to check for sliding323 or hybrid layer structure, and initialize each layer accordingly.324 max_cache_len (`int`):325 The maximum number of tokens that this Cache should hold.326 offloading (`bool`, *optional*, defaults to `False`):327 Whether to perform offloading of the layers to `cpu`, to save GPU memory.328 offload_only_non_sliding (`bool`, *optional*, defaults to `True`):329 If `offloading` is `True`, this further decides if only the non-sliding layers will be offloaded (because330 usually the sliding layers are small in size, so there is no need to offload them, and skipping it is faster).331 332 Example:333 334 ```python335 >>> from transformers import AutoTokenizer, AutoModelForCausalLM, StaticCache336 337 >>> model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf")338 >>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")339 340 >>> inputs = tokenizer(text="My name is Llama", return_tensors="pt")341 342 >>> # Prepare a cache class and pass it to model's forward343 >>> # Leave empty space for 10 new tokens, which can be used when calling forward iteratively 10 times to generate344 >>> max_generated_length = inputs.input_ids.shape[1] + 10345 >>> past_key_values = StaticCache(config=model.config, max_cache_len=max_generated_length)346 >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True)347 >>> outputs.past_key_values # access cache filled with key/values from generation348 StaticCache()349 ```350 FTr� rr r� r� c 351 s |j dd�}t|dd �}|d u r@t|dd �d ur#dd� t|j�D �}n t|dd �d ur6dd� t|j�D �}n352d d� t|j�D �}t|d353�rM|d |j � }g }|D ]$}|d kr_t||jd �} n|d
krkt||jd �} nt |d�} |�354| � qQt � j |||d� d S )NTr� r� r_ c S � g | ]}d �qS )r� r r� r r r r� < � z(StaticCache.__init__.<locals>.<listcomp>r� c S r� )r� r r� r r r r� > r� c S r� )r� r r� r r r r� @ r� r� r� )rr r_ r� r~ r� )
r� r� r� r� r3 r� r� r_ r� rq r� ra r )355r r� rr r� r� �kwargsr� r� r� r� rb r r r / s* 356�357 zStaticCache.__init__)FT) r r9 r: r; r rB r� r rp r r rb r r� s +�����r� c sJ e Zd ZdZ ddededed ed358ed ed ef� fd
d�
Z� ZS )�QuantizedCachea� 359 A quantizer cache similar to what is described in the360 [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache paper](https://huggingface.co/papers/2402.02750).361 It allows the model to generate longer sequence length without allocating too much memory for keys and values362 by applying quantization.363 The cache has two types of storage, one for original precision and one for the364 quantized cache. A `residual length` is set as a maximum capacity for the original precision cache. When the365 length goes beyond maximum capacity, the original precision cache is discarded and moved into the quantized cache.366 The quantization is done per-channel with a set `q_group_size` for both keys and values, in contrast to what was367 described in the paper.368 369 See `Cache` for details on common methods that are implemented by all cache classes.370 371 Args:372 backend (`str`):373 The quantization backend to use. One of `("quanto", "hqq").374 config (`PretrainedConfig`):375 The config of the model for which this Cache will be used.376 nbits (`int`, *optional*, defaults to 4):377 The number of bits for quantization.378 axis_key (`int`, *optional*, defaults to 0):379 The axis on which to quantize the keys.380 axis_value (`int`, *optional*, defaults to 0):381 The axis on which to quantize the values.382 q_group_size (`int`, *optional*, defaults to 64):383 Quantization is done per-channel according to a set `q_group_size` for both keys and values.384 residual_length (`int`, *optional*, defaults to 128):385 Maximum capacity for the original precision cache386 r� r r� r� �backendr� r� r� r� r� r� c sl |dkrt �n|dkrt�ntd|� d���|jdd�}� �����fdd�t|j�D �}t� j|d � d S )387N�quanto�hqqz Unknown quantization backend `�`Tr� c s g | ]388}��� �����qS r r r� �r� r� Z layer_classr� r� r� r r r� � r� z+QuantizedCache.__init__.<locals>.<listcomp>r� )r� r� ro r� r� r� ra r ) r r r� r� r� r� r� r� r� rb r r r u s 389 �zQuantizedCache.__init__r� ) r r9 r: r; r@ r rB r rp r r rb r r� V s, "��������r� c @ sf e Zd ZdZd7dd�Zdefdd�Zdd � Zd390ede e391j e392j e393j e394j f fd d �Z d
d� Z
de e e395j fdd�Zedeee e396jdf dd fdd��Zd8d397edefdd�Zdd� Zde398jfdd �Zd efd d �Zd efd!d"�Zd#ed$edd%fd&d'�Zd(efd)d*�Zd+e399j fd,d-�Z defd.d/�Z d0e400j d401ede eef fd1d2�Z e d3d4� �Z e de!fd5d6��Z"dS )9�EncoderDecoderCachea� 402 Base, abstract class for all encoder-decoder caches. Can be used to hold combinations of self-attention and403 cross-attention caches.404 405 See `Cache` for details on common methods that are implemented by all cache classes.406 407 Args:408 caches (`Iterable`):409 Usually an iterable of length 2, containing 2 `Cache` objects, the first one for self-attention, the410 second one for cross-attention. Can optionally also be an iterable of length 1, containing a411 `tuple[tuple[torch.Tensor]]` (usually used for compatibility with torch dp and ddp).412 413 Example:414 415 ```python416 >>> from transformers import AutoProcessor, AutoModelForCausalLM, DynamicCache, EncoderDecoderCache417 418 >>> model = AutoModelForCausalLM.from_pretrained("openai/whisper-small")419 >>> processor = AutoProcessor.from_pretrained("openai/whisper-small")420 421 >>> inputs = processor(audio=YOUR-AUDIO, return_tensors="pt")422 423 >>> # Prepare cache classes for encoder and decoder and pass it to model's forward424 >>> self_attention_cache = DynamicCache(config=self.config)425 >>> cross_attention_cache = DynamicCache(config=self.config)426 >>> past_key_values = EncoderDecoderCache(self_attention_cache, cross_attention_cache)427 >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True)428 >>> outputs.past_key_values # access cache filled with key/values from generation429 EncoderDecoderCache()430 ```431 r% Nc G s, t |�dkr@t� | _t� | _t|d �D ]*\}}|d d� \}}| j�|||� t |�dkr>|dd � \}}| j�|||� qn:t |�dkrqt|d t�rTt|d t�sftdt |d ��dt |d �����|d | _|d | _n t432dt |�� ���i | _ t t | j��D ]}t
| j�|�dk�| j |< q�d S )Nr r ry z;One of the two arguments is not a Cache: type(caches[0]) = z, type(caches[1]) = z Expected 1 or 2 arguments, got )r� r� �self_attention_cache�cross_attention_cacher� r' �433isinstancer� � TypeErrorr� ro �434is_updatedr� r� r* )r �cachesr� �key_value_statesr r# r r r r � s* �� $435 �z EncoderDecoderCache.__init__c C s | j j� d| j� d| j� d�S )Nz(self_attention_cache=z, cross_attention_cache=r� )r r r r r r r r r � s ��z EncoderDecoderCache.__repr__c c sN � t t| ��D ] }| jj| j| jj| j| jj| j| jj| jfV qdS r� )r� r� r r� r r r r� r r r r� � s � ��z EncoderDecoderCache.__iter__r� c C sX |t | �k r | jj| j| jj| j| jj| j| jj| jfS tdt | �� d|� ���r� )r� r r� r r r r� r� r r r r� � s �z EncoderDecoderCache.__getitem__c C r� )z�436 Support for backwards-compatible `past_key_values` length, e.g. `len(past_key_values)`. This value corresponds437 to the number of layers in the model.438 )r� r r r r r r� � s 439zEncoderDecoderCache.__len__c C sR d}t | j�dkr"t| j�� | j�� �D ] \}}||| f7 }q|S | j�� }|S )z[Converts the `EncoderDecoderCache` instance into its equivalent in the legacy cache format.r r )r� r �zipr r� )r r� � self_attn�440cross_attnr r r r� � s �441�z#EncoderDecoderCache.to_legacy_cacher� .c C s� | t � t � �}|du rt�d� |S t|�D ]/\}}|dd� \}}|j�|||� t|�dkrE|dd� \}}|j�|||� d|j|< q|S )zUConverts a cache in the legacy cache format into an equivalent `EncoderDecoderCache`.Nr� ry T) r� r� r� r� r r' r� r r442 )r� r� r� r� r r r# r r r r� � s 443 � 444�z%EncoderDecoderCache.from_legacy_cacher c C s | j �|�S )zYReturns the sequence length of the cached states. A layer index can be optionally passed.)r r* r� r r r r* s z"EncoderDecoderCache.get_seq_lengthc C s. | j �� | j�� | jD ]}d| j|< q
d S r )r r4 r r445 r� r r r r4 s446 447448449 �zEncoderDecoderCache.resetr5 c C s | j �|� | j�|� dS )zDReorders the cache for beam search, given the selected beam indices.N)r r7 r r6 r r r r7 s z!EncoderDecoderCache.reorder_cache�methodc C sD t | jt�r t | jt�s td|� d| j�� � d| j�� � d���d S )Nr z)` is only defined for dynamic cache, got z" for the self attention cache and z for the cross attention cache.)r r r� r ro �__str__)r r r r r �check_dynamic_cache s 450�451����z'EncoderDecoderCache.check_dynamic_cache�maximum_lengthc C s | � | jj� | j�|� dS )z�452 Crop the past key values up to a new `maximum_length` in terms of tokens. `maximum_length` can also be453 negative to remove `maximum_length` tokens. This is used in assisted decoding and contrastive search (on the Hub).454 N)r rV r r )r r r r r rV * s zEncoderDecoderCache.crop�full_batch_size�455split_sizezlist[EncoderDecoderCache]c C sV | � | jj� | j�||�}| j�||�}g }t||�D ] \}}|�t||�� q |S )z�456 Split the current instance into a list of `DynamicCache` by the batch size. This will be used by457 `_split_model_inputs()` in `generation.utils`458 )r � batch_splitr r r r
r� r )r r r r r �outr r r r r r 2 s z EncoderDecoderCache.batch_splitrW c C �* | � | jj� | j�|� | j�|� dS )zaRepeat the cache `repeats` times in the batch dimension. Used in contrastive search (on the Hub).N)r rY r r r rX r r r rY @ � z+EncoderDecoderCache.batch_repeat_interleaverZ c C r )zeOnly keep the `indices` in the batch dimension of the cache. Used in contrastive search (on the Hub).N)r r\ r r r r[ r r r r\ F r z(EncoderDecoderCache.batch_select_indicesc C s459 | j �� S )zKReturns the maximum sequence length (i.e. max capacity) of the cache object)r r+ r r r r r+ L s 460z'EncoderDecoderCache.get_max_cache_shaper( c C s | j �||�S r )r r) r� r r r r) P s z"EncoderDecoderCache.get_mask_sizesc C � | j jS r )r r] r r r r r] S � z EncoderDecoderCache.is_slidingc C r r )r r< r r r r r< W r z"EncoderDecoderCache.is_compileabler8 r� )#r r9 r: r; r r@ r r� rB rA r= r> r� r� r� r� r r Z FloatTensorr� r* r4 rC r7 r rV r rY r\ r+ r) r� r] r� r< r r r r r � s8 461 &
� � 462r c �& e Zd Zdedef� fdd� Z� ZS )�SlidingWindowLayerrr r_ c � t �d� t� �||� d S )Nz�`SlidingWindowLayer` is deprecated and will be removed in version v4.59 Use `StaticSlidingWindowLayer` instead, which is a better name for it.�r� r� ra r �r rr r_ rb r r r ` � �zSlidingWindowLayer.__init__�r r9 r: rB r rp r r rb r r _ � r c r )�ChunkedSlidingLayerrr r_ c r )Nz�`ChunkedSlidingLayer` is deprecated and will be removed in version v4.59 Use `StaticSlidingWindowLayer` instead, which has the exact same functionalities.r r rb r r r i r! z ChunkedSlidingLayer.__init__r" r r rb r r$ h r# r$ c s e Zd Zd� fdd� Z� ZS )�OffloadedCacher% Nc s t �d� t� jdd� d S )Nzo`OffloadedCache` is deprecated and will be removed in version v4.59 Use `DynamicCache(offloading=True)` insteadT)r� r r rb r r r r r! zOffloadedCache.__init__r8 )r r9 r: r rp r r rb r r% q s r% c �& e Zd Zdedef� fdd� Z� ZS )�OffloadedStaticCacher� rr c � t �d� t� j||dd� d S )Nzy`OffloadedStaticCache` is deprecated and will be removed in version v4.59 Use `StaticCache(..., offloading=True)` insteadT�r� rr r� r �r r� rr �argsr� rb r r r { � �z OffloadedStaticCache.__init__�r r9 r: r rB r rp r r rb r r' z r# r' c r& )�SlidingWindowCacher� rr c � t �d� t� j||d� d S )Nz�`SlidingWindowCache` is deprecated and will be removed in version v4.59 Use `StaticCache(...)` instead which will correctly infer the type of each layer.�r� rr r r* rb r r r � � �zSlidingWindowCache.__init__r- r r rb r r. � r# r. c r& )� HybridCacher� rr c r/ )Nz�`HybridCache` is deprecated and will be removed in version v4.59 Use `StaticCache(...)` instead which will correctly infer the type of each layer.r0 r r* rb r r r � r1 zHybridCache.__init__r- r r rb r r2 � r# r2 c r& )�HybridChunkedCacher� rr c r/ )Nz�`HybridChunkedCache` is deprecated and will be removed in version v4.59 Use `StaticCache(...)` instead which will correctly infer the type of each layer.r0 r r* rb r r r � r1 zHybridChunkedCache.__init__r- r r rb r r3 � r# r3 c r& )�OffloadedHybridCacher� rr c r( )Nz�`OffloadedHybridCache` is deprecated and will be removed in version v4.59 Use `StaticCache(..., offload=True)` instead which will correctly infer the type of each layer.Tr) r r* rb r r r � r, z OffloadedHybridCache.__init__r- r r rb r r4 � r# r4 c
�B e Zd Z d
dedededed ed463ef � fd d �
Z� ZS )�QuantoQuantizedCacher� r r� r� r� r� r� r� r� r� c �& t �d� t� �d||||||� d S )Nz~`QuantoQuantizedCache` is deprecated and will be removed in version v4.59 Use `QuantizedCache(backend='quanto', ...)` instead.r r �r r� r� r� r� r� r� rb r r r � � � z QuantoQuantizedCache.__init__r� r- r r rb r r6 � �& �������r6 c
r5 )�HQQQuantizedCacher� r r� r� r� r� r� r� r� r� c r7 )Nzx`HQQQuantizedCache` is deprecated and will be removed in version v4.59 Use `QuantizedCache(backend='hqq', ...)` instead.r r r8 rb r r r � r9 zHQQQuantizedCache.__init__r� r- r r rb r r; � r: r; c @ s e Zd ZdZddd�ZdS )� SinkCachea
464 It is now a `custom_generate` repository on the Hub: https://huggingface.co/transformers-community/sink_cache.465 See [these docs](https://huggingface.co/docs/transformers/generation_strategies#custom-decoding-methods) for466 general `custom_generate`usage.467 r% Nc K s t d��)Nz�`SinkCache` has been moved as a `custom_generate` repository on the Hub: https://huggingface.co/transformers-community/sink_cache. See the repository for usage examples.)r} )r r� r r r r � s �zSinkCache.__init__r8 )r r9 r: r; r r r r r r< � s r< )0�abcr r �collections.abcr �typingr r r= Zconfiguration_utilsr �utilsr r468 r r r
Zhqq.core.quantizer r� r� Z469get_loggerr r� r rD r^ rq r� r� r� r� r� r� r� r� r r r$ r% r' r. r2 r3 r4 r6 r; r< r r r r �<module> sH 470:SQm}P26 uyO8 R