declare-lab/tango2
92
1a
2 ��'d�f � @ s� d dl Z d dlmZ d dlmZmZmZmZmZ d dl Z ddl3m Z ddl m
Z
mZmZmZmZmZmZ e� rvd dlZe� r�d dlmZmZ e�e�ZdZd Z d4Z d Z G d d
� d
e j j �Z!G dd� d�Z"G dd� d�Z#dS )� N)� defaultdict)�Callable�Dict�List�Optional�Union� )�LoRAAttnProcessor)�DIFFUSERS_CACHE�HF_HUB_OFFLINE�_get_model_file� deprecate�is_safetensors_available�is_transformers_available�logging)�PreTrainedModel�PreTrainedTokenizerzpytorch_lora_weights.binz pytorch_lora_weights.safetensorszlearned_embeds.binzlearned_embeds.safetensorsc s, e Zd Zeeejf d�� fdd� Z� ZS )�AttnProcsLayers)�5state_dictc sr t � �� tj�|�� �| _tt|� � ��| _6dd� t|� � �D �| _ dd� }dd� }| � |� | j
|dd� d S ) Nc S s i | ]\}}||�qS � r ��.0�k�vr r �F/home/deep/Projects/audio_diffusion/diffusers/src/diffusers/loaders.py�7<dictcomp>6 � z,AttnProcsLayers.__init__.<locals>.<dictcomp>c _ sL i }|� � D ]:\}}t|�d�d �}|�d|� �| j| �}|||< q |S )N�.r �layers.)�items�int�split�replace�mapping) �moduler �args�kwargsZnew_state_dict�key�value�num�new_keyr r r �map_to: s 8z(AttnProcsLayers.__init__.<locals>.map_toc _ sV t |�� �}|D ]@}|�d�d d }|�|d| j| � ��}|| ||< ||= qd S )Nz9.processorr r )�list�keysr! r" � rev_mapping)r$ r r% r& �all_keysr' Z replace_keyr* r r r �map_fromC s z*AttnProcsLayers.__init__.<locals>.map_fromT)� with_module)�super�__init__�torch�nn�10ModuleList�values�layers�dict� enumerater- r# r. �_register_state_dict_hook�"_register_load_state_dict_pre_hook)�selfr r+ r0 �� __class__r r r3 2 s 11 12zAttnProcsLayers.__init__) �__name__�13__module__� __qualname__r �strr4 �Tensorr3 �
__classcell__r r r> r r 1 s r c @ sN e Zd Zeeeeejf f d�dd�Zd14eee j15f e ee e d�dd �Z
dS ) �UNet2DConditionLoadersMixin)�%pretrained_model_name_or_path_or_dictc
s� |� dt�}|� dd�}|� dd�}|� dd�}|� dt�}|� dd�}|� d d�} |� d16d�}17|� d d�} |� d d�} | r�t� s�td
��d}
| du r�t� } d}
ddd�}d}t|t��s�| r�| du s�| du�rD| �d��rDz6t|| p�t ||||||| |18|d� }t19j j |dd�}W n0 t
�yB } z|
�s.|�W Y d}~n20d}~0 0 |du �r�t|| �pZt||||||| |21|d� }t j|dd�}n|}i }tdd� |�� D ��}|�r`tt�}|�� D ]H\}}d�|�d�dd� �d�|�d�dd� � }}||| |< �q�|�� D ]T\}}|d jd }|d jd }|d jd } t| ||d �||< || �|� �qnt|� d ���� fd!d"�|�� D �}� �|� dS )#a3 22 Load pretrained attention processor layers into `UNet2DConditionModel`. Attention processor layers have to be23 defined in24 [cross_attention.py](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/cross_attention.py)25 and be a `torch.nn.Module` class.26 27 <Tip warning={true}>28 29 This function is experimental and might change in the future.30 31 </Tip>32 33 Parameters:34 pretrained_model_name_or_path_or_dict (`str` or `os.PathLike` or `dict`):35 Can be either:36 37 - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co.38 Valid model ids should have an organization name, like `google/ddpm-celebahq-256`.39 - A path to a *directory* containing model weights saved using [`~ModelMixin.save_config`], e.g.,40 `./my_model_directory/`.41 - A [torch state42 dict](https://pytorch.org/tutorials/beginner/saving_loading_models.html#what-is-a-state-dict).43 44 cache_dir (`Union[str, os.PathLike]`, *optional*):45 Path to a directory in which a downloaded pretrained model configuration should be cached if the46 standard cache should not be used.47 force_download (`bool`, *optional*, defaults to `False`):48 Whether or not to force the (re-)download of the model weights and configuration files, overriding the49 cached versions if they exist.50 resume_download (`bool`, *optional*, defaults to `False`):51 Whether or not to delete incompletely received files. Will attempt to resume the download if such a52 file exists.53 proxies (`Dict[str, str]`, *optional*):54 A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128',55 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request.56 local_files_only(`bool`, *optional*, defaults to `False`):57 Whether or not to only look at local files (i.e., do not try to download the model).58 use_auth_token (`str` or *bool*, *optional*):59 The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated60 when running `diffusers-cli login` (stored in `~/.huggingface`).61 revision (`str`, *optional*, defaults to `"main"`):62 The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a63 git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any64 identifier allowed by git.65 subfolder (`str`, *optional*, defaults to `""`):66 In case the relevant files are located inside a subfolder of the model repo (either remote in67 huggingface.co or downloaded locally), you can specify the folder name here.68 69 mirror (`str`, *optional*):70 Mirror source to accelerate downloads in China. If you are from China and have an accessibility71 problem, you can set this option to resolve it. Note that we do not guarantee the timeliness or safety.72 Please refer to the mirror site for more information.73 74 <Tip>75 76 It is required to be logged in (`huggingface-cli login`) when you want to use private or [gated77 models](https://huggingface.co/docs/hub/models-gated#gated-models).78 79 </Tip>80 � cache_dir�force_downloadF�resume_download�proxiesN�local_files_only�use_auth_token�revision� subfolder� weight_name�use_safetensors�r`use_safetensors`=True but safetensors is not installed. Please install safetensors with `pip install safetenstorsTZattn_procs_weights�pytorch�� file_type� framework� .safetensors�81� weights_namerH rI rJ rK rL rM rN rO �82user_agent�cpu��device�� map_locationc s s | ]}d |v V qdS )ZloraNr )r r r r r � <genexpr>� r z>UNet2DConditionLoadersMixin.load_attn_procs.<locals>.<genexpr>r �����zto_k_lora.down.weightr r zto_k_lora.up.weight)� hidden_size�cross_attention_dim�rankzE does not seem to be in the correct format expected by LoRA training.c s$ i | ] \}}||j � j� jd ��qS ))r] �dtype)�tor] re r �r= r r r � r z?UNet2DConditionLoadersMixin.load_attn_procs.<locals>.<dictcomp>)�popr83 r r �84ValueError�85isinstancer9 �endswithr �LORA_WEIGHT_NAME_SAFE� safetensorsr4 � load_file�IOError�LORA_WEIGHT_NAME�load�allr- r r �joinr! �shaper �load_state_dict�set_attn_processor) r= rG r& rH rI rJ rK rL rM rN rO rP rQ � allow_picklerZ �86model_filer �e�attn_processors�is_loraZlora_grouped_dictr' r( Zattn_processor_key�sub_key�87value_dictrd rc rb r rg r �load_attn_procsP s� > 88�� ���
89�
2�90z+UNet2DConditionLoadersMixin.load_attn_procsTNF)�save_directory�is_main_processrP �
save_function�safe_serializationc K s� |pt ddd|d�}tj�|�r6t�d|� d�� dS |du rR|rLdd � }ntj}tj|d91d � t | j92�}|� � }|du r�|r�t }nt
}||tj�||�� t�d tj�||�� �� dS )
a� 93 Save an attention processor to a directory, so that it can be re-loaded using the94 `[`~loaders.UNet2DConditionLoadersMixin.load_attn_procs`]` method.95 96 Arguments:97 save_directory (`str` or `os.PathLike`):98 Directory to which to save. Will be created if it doesn't exist.99 is_main_process (`bool`, *optional*, defaults to `True`):100 Whether the process calling this is the main process or not. Useful when in distributed training like101 TPUs and need to call this function on all processes. In this case, set `is_main_process=True` only on102 the main process to avoid race conditions.103 save_function (`Callable`):104 The function to use to save the state dictionary. Useful on distributed training like TPUs when one105 need to replace `torch.save` by another method. Can be configured with the environment variable106 `DIFFUSERS_SAVE_MODE`.107 rY z0.18.0z?`weights_name` is deprecated, please use `weight_name` instead.)� take_fromzProvided path (z#) should be a directory, not a fileNc S s t jj| |ddid�S )N�format�pt)�metadata)rm r4 � save_file)�weights�filenamer r r r� s zBUNet2DConditionLoadersMixin.save_attn_procs.<locals>.save_functionT)�exist_okzModel weights saved in )r
�os�path�isfile�logger�errorr4 �save�makedirsr rz r rl rp rs �info) r= r r� rP r� r� r& �
model_to_saver r r r �save_attn_procs� s, � 108109z+UNet2DConditionLoadersMixin.save_attn_procs)TNNF)r@ rA rB r rC r r4 rD r~ r� �PathLike�boolr r� r r r r rF O s & � �rF c @ sd e Zd ZdZeeee f dd�dd�Zedd�dd�Zd eee ee110j f f e e d �d111d �Z
dS )
�TextualInversionLoaderMixinzl112 Mixin class for loading textual inversion tokens and embeddings to the tokenizer and text encoder.113 r )�prompt� tokenizerc s@ t |t�s|g}n|}� �fdd�|D �}t |t�s<|d S |S )a9 114 Maybe convert a prompt into a "multi vector"-compatible prompt. If the prompt includes a token that corresponds115 to a multi-vector textual inversion embedding, this function will process the prompt so that the special token116 is replaced with multiple special tokens each corresponding to one of the vectors. If the prompt has no textual117 inversion token or a textual inversion token that is a single vector, the input prompt is simply returned.118 119 Parameters:120 prompt (`str` or list of `str`):121 The prompt or prompts to guide the image generation.122 tokenizer (`PreTrainedTokenizer`):123 The tokenizer responsible for encoding the prompt into input tokens.124 125 Returns:126 `str` or list of `str`: The converted prompt127 c s g | ]}� � |���qS r )�_maybe_convert_prompt)r �p�r= r� r r �128<listcomp>J r zDTextualInversionLoaderMixin.maybe_convert_prompt.<locals>.<listcomp>r )rj r )r= r� r� �promptsr r� r �maybe_convert_prompt5 s 129130z0TextualInversionLoaderMixin.maybe_convert_promptc C sf |� |�}|D ]R}||jv r|}d}|� d|� �|jv rT||� d|� �7 }|d7 }q$|�||�}q|S )a 131 Maybe convert a prompt into a "multi vector"-compatible prompt. If the prompt includes a token that corresponds132 to a multi-vector textual inversion embedding, this function will process the prompt so that the special token133 is replaced with multiple special tokens each corresponding to one of the vectors. If the prompt has no textual134 inversion token or a textual inversion token that is a single vector, the input prompt is simply returned.135 136 Parameters:137 prompt (`str`):138 The prompt to guide the image generation.139 tokenizer (`PreTrainedTokenizer`):140 The tokenizer responsible for encoding the prompt into input tokens.141 142 Returns:143 `str`: The converted prompt144 r �_)�tokenize�added_tokens_encoderr" )r= r� r� �tokens�token� replacement�ir r r r� Q s 145146147z1TextualInversionLoaderMixin._maybe_convert_promptN)� pretrained_model_name_or_pathr� c
s t | d�rt| jt�s2t| jj� d| jj� d���t | d�rHt| jt �sdt| jj� d| jj� d���|�148dt �}|�149dd�}|�150d d�}|�151d152d �}|�153d t �}|�154d
d �} |�155dd �}156|�157dd �} |�158dd �} |�159dd �}
|
r�t
� s�td��d}|
d u �rt
� }
d}ddd�}d }|
�r$| d u �s:| d u�r�| �d��r�z8t|| �pHt|||||| |160| |d� }tjj|dd�}W n4 t�y� } z|�s�|�d }W Y d }~n161d }~0 0 |d u �r�t|| �p�t|||||| |162| |d� }tj|dd�}t|tj��r163� d u �rtd ��|}nBt|�d k�r.tt|�� ��\}}n d |v �rL|d }|d d }� d u�rz|� k�rzt � d!|� d"� � d#�� n|� |j | jj | jj d$�}| j�!� }� |v �r�td%� � d&� � d'���nh� � d(�|v �r(� g}d }� � d)|� �| jj"v �r|�#� � d)|� �� |d 7 }�q�td*|� d+|� d'���t|j$�d k�oD|j$d, d k}|�r�� g� fd-d.�t%d |j$d, �D � }d/d.� |D �}n&� g}t|j$�d k�r�|d, gn|g}| j�&|� | j�'|�}| j�(t| j�� t)||�D ] \} }|| j�*� j+j,| < �q�t �-d0� d S )1a� 164 Load textual inversion embeddings into the text encoder of stable diffusion pipelines. Both `diffusers` and165 `Automatic1111` formats are supported.166 167 <Tip warning={true}>168 169 This function is experimental and might change in the future.170 171 </Tip>172 173 Parameters:174 pretrained_model_name_or_path (`str` or `os.PathLike`):175 Can be either:176 177 - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co.178 Valid model ids should have an organization name, like179 `"sd-concepts-library/low-poly-hd-logos-icons"`.180 - A path to a *directory* containing textual inversion weights, e.g.181 `./my_text_inversion_directory/`.182 weight_name (`str`, *optional*):183 Name of a custom weight file. This should be used in two cases:184 185 - The saved textual inversion file is in `diffusers` format, but was saved under a specific weight186 name, such as `text_inv.bin`.187 - The saved textual inversion file is in the "Automatic1111" form.188 cache_dir (`Union[str, os.PathLike]`, *optional*):189 Path to a directory in which a downloaded pretrained model configuration should be cached if the190 standard cache should not be used.191 force_download (`bool`, *optional*, defaults to `False`):192 Whether or not to force the (re-)download of the model weights and configuration files, overriding the193 cached versions if they exist.194 resume_download (`bool`, *optional*, defaults to `False`):195 Whether or not to delete incompletely received files. Will attempt to resume the download if such a196 file exists.197 proxies (`Dict[str, str]`, *optional*):198 A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128',199 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request.200 local_files_only(`bool`, *optional*, defaults to `False`):201 Whether or not to only look at local files (i.e., do not try to download the model).202 use_auth_token (`str` or *bool*, *optional*):203 The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated204 when running `diffusers-cli login` (stored in `~/.huggingface`).205 revision (`str`, *optional*, defaults to `"main"`):206 The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a207 git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any208 identifier allowed by git.209 subfolder (`str`, *optional*, defaults to `""`):210 In case the relevant files are located inside a subfolder of the model repo (either remote in211 huggingface.co or downloaded locally), you can specify the folder name here.212 213 mirror (`str`, *optional*):214 Mirror source to accelerate downloads in China. If you are from China and have an accessibility215 problem, you can set this option to resolve it. Note that we do not guarantee the timeliness or safety.216 Please refer to the mirror site for more information.217 218 <Tip>219 220 It is required to be logged in (`huggingface-cli login`) when you want to use private or [gated221 models](https://huggingface.co/docs/hub/models-gated#gated-models).222 223 </Tip>224 r� zF requires `self.tokenizer` of type `PreTrainedTokenizer` for calling `�`� text_encoderzE requires `self.text_encoder` of type `PreTrainedModel` for calling `rH rI FrJ rK NrL rM rN rO rP rQ rR TZtext_inversionrS rT rW rX r[ r\ r^ z�You are trying to load a textual inversion embedding that has been saved as a PyTorch tensor. Make sure to pass the name of the corresponding token in this case: `token=...`.r Zstring_to_param�name�*zThe loaded token: z$ is overwritten by the passed token r )re r] zToken zQ already in tokenizer vocabulary. Please choose a different token name or remove z3 and embedding from the tokenizer and text encoder.Z_1r� zMulti-vector Token zU already in tokenizer vocabulary. Please choose a different token name or remove the r c s g | ]}� � d |� ��qS )r� r )r r� �r� r r r� * r zFTextualInversionLoaderMixin.load_textual_inversion.<locals>.<listcomp>c S s g | ]}|�qS r r )r ry r r r r� + r z/Loaded textual inversion embedding for {token}.).�hasattrrj r� r ri r? r@ �load_textual_inversionr� r rh r225 r r rk r �TEXT_INVERSION_NAME_SAFErm r4 rn � Exception�TEXT_INVERSION_NAMErq rD �len�next�iterr r� �warnrf re r] Z get_vocabr� �appendrt �rangeZ226add_tokensZconvert_tokens_to_idsZresize_token_embeddings�zipZget_input_embeddings�weight�datar� ) r= r� r� r& rH rI rJ rK rL rM rN rO rP rQ rw rZ rx r ry � embeddingZ loaded_tokenZvocabZmulti_vector_tokensr� Zis_multi_vectorr� �227embeddingsZ token_idsZtoken_idr r� r r� n s� A228��229�� 230�231����
232�
233�234 235236� � $ z2TextualInversionLoaderMixin.load_textual_inversion)N)r@ rA rB �__doc__r rC r r� r� r r4 rD r r� r r r r r� 0 s ��r� )$r� � collectionsr �typingr r r r r r4 Zmodels.attention_processorr �utilsr237 r r r
r r r rm � transformersr r �238get_loggerr@ r� rp rl r� r� r5 �Moduler rF r� r r r r �<module> s$ $ 239 b