diffusers/community-pipelines-mirror

1# Community Scripts2 3**Community scripts** consist of inference examples using Diffusers pipelines that have been added by the community.4Please have a look at the following table to get an overview of all community examples. Click on the **Code Example** to get a copy-and-paste code example that you can try out.5If a community script doesn't work as expected, please open an issue and ping the author on it.6 7| Example                                                                                                                               | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | Code Example                                                                              | Colab                                                                                                                                                                                                              |                                                        Author |8|:--------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------:|9| Using IP-Adapter with Negative Noise                                                                                                  | Using negative noise with IP-adapter to better control the generation (see the [original post](https://github.com/huggingface/diffusers/discussions/7167) on the forum for more details)                                                                                                                                                                                                                                                    | [IP-Adapter Negative Noise](#ip-adapter-negative-noise)                                   |[Notebook](https://github.com/huggingface/notebooks/blob/main/diffusers/ip_adapter_negative_noise.ipynb) | [Álvaro Somoza](https://github.com/asomoza)|10| Asymmetric Tiling                                                                                                  |configure seamless image tiling independently for the X and Y axes                                                                                                                                                                                                      | [Asymmetric Tiling](#Asymmetric-Tiling )                                   |[Notebook](https://github.com/huggingface/notebooks/blob/main/diffusers/asymetric_tiling.ipynb) | [alexisrolland](https://github.com/alexisrolland)|11| Prompt Scheduling Callback                                                                                                  |Allows changing prompts during a generation                                                                                                                                                                                                      | [Prompt Scheduling-Callback](#Prompt-Scheduling-Callback )                                   |[Notebook](https://github.com/huggingface/notebooks/blob/main/diffusers/prompt_scheduling_callback.ipynb) | [hlky](https://github.com/hlky)|12 13 14## Example usages15 16### IP Adapter Negative Noise17 18Diffusers pipelines are fully integrated with IP-Adapter, which allows you to prompt the diffusion model with an image. However, it does not support negative image prompts (there is no `negative_ip_adapter_image` argument) the same way it supports negative text prompts. When you pass an `ip_adapter_image,` it will create a zero-filled tensor as a negative image. This script shows you how to create a negative noise from `ip_adapter_image` and use it to significantly improve the generation quality while preserving the composition of images.19 20[cubiq](https://github.com/cubiq) initially developed this feature in his [repository](https://github.com/cubiq/ComfyUI_IPAdapter_plus). The community script was contributed by [asomoza](https://github.com/Somoza). You can find more details about this experimentation [this discussion](https://github.com/huggingface/diffusers/discussions/7167)21 22IP-Adapter without negative noise23|source|result|24|---|---|25|![20240229150812](https://github.com/huggingface/diffusers/assets/5442875/901d8bd8-7a59-4fe7-bda1-a0e0d6c7dffd)|![20240229163923_normal](https://github.com/huggingface/diffusers/assets/5442875/3432e25a-ece6-45f4-a3f4-fca354f40b5b)|26 27IP-Adapter with negative noise28|source|result|29|---|---|30|![20240229150812](https://github.com/huggingface/diffusers/assets/5442875/901d8bd8-7a59-4fe7-bda1-a0e0d6c7dffd)|![20240229163923](https://github.com/huggingface/diffusers/assets/5442875/736fd15a-36ba-40c0-a7d8-6ec1ac26f788)|31 32```python33import torch34 35from diffusers import AutoencoderKL, DPMSolverMultistepScheduler, StableDiffusionXLPipeline36from diffusers.models import ImageProjection37from diffusers.utils import load_image38 39 40def encode_image(41    image_encoder,42    feature_extractor,43    image,44    device,45    num_images_per_prompt,46    output_hidden_states=None,47    negative_image=None,48):49    dtype = next(image_encoder.parameters()).dtype50 51    if not isinstance(image, torch.Tensor):52        image = feature_extractor(image, return_tensors="pt").pixel_values53 54    image = image.to(device=device, dtype=dtype)55    if output_hidden_states:56        image_enc_hidden_states = image_encoder(image, output_hidden_states=True).hidden_states[-2]57        image_enc_hidden_states = image_enc_hidden_states.repeat_interleave(num_images_per_prompt, dim=0)58 59        if negative_image is None:60            uncond_image_enc_hidden_states = image_encoder(61                torch.zeros_like(image), output_hidden_states=True62            ).hidden_states[-2]63        else:64            if not isinstance(negative_image, torch.Tensor):65                negative_image = feature_extractor(negative_image, return_tensors="pt").pixel_values66            negative_image = negative_image.to(device=device, dtype=dtype)67            uncond_image_enc_hidden_states = image_encoder(negative_image, output_hidden_states=True).hidden_states[-2]68 69        uncond_image_enc_hidden_states = uncond_image_enc_hidden_states.repeat_interleave(num_images_per_prompt, dim=0)70        return image_enc_hidden_states, uncond_image_enc_hidden_states71    else:72        image_embeds = image_encoder(image).image_embeds73        image_embeds = image_embeds.repeat_interleave(num_images_per_prompt, dim=0)74        uncond_image_embeds = torch.zeros_like(image_embeds)75 76        return image_embeds, uncond_image_embeds77 78 79@torch.no_grad()80def prepare_ip_adapter_image_embeds(81    unet,82    image_encoder,83    feature_extractor,84    ip_adapter_image,85    do_classifier_free_guidance,86    device,87    num_images_per_prompt,88    ip_adapter_negative_image=None,89):90    if not isinstance(ip_adapter_image, list):91        ip_adapter_image = [ip_adapter_image]92 93    if len(ip_adapter_image) != len(unet.encoder_hid_proj.image_projection_layers):94        raise ValueError(95            f"`ip_adapter_image` must have same length as the number of IP Adapters. Got {len(ip_adapter_image)} images and {len(unet.encoder_hid_proj.image_projection_layers)} IP Adapters."96        )97 98    image_embeds = []99    for single_ip_adapter_image, image_proj_layer in zip(100        ip_adapter_image, unet.encoder_hid_proj.image_projection_layers101    ):102        output_hidden_state = not isinstance(image_proj_layer, ImageProjection)103        single_image_embeds, single_negative_image_embeds = encode_image(104            image_encoder,105            feature_extractor,106            single_ip_adapter_image,107            device,108            1,109            output_hidden_state,110            negative_image=ip_adapter_negative_image,111        )112        single_image_embeds = torch.stack([single_image_embeds] * num_images_per_prompt, dim=0)113        single_negative_image_embeds = torch.stack([single_negative_image_embeds] * num_images_per_prompt, dim=0)114 115        if do_classifier_free_guidance:116            single_image_embeds = torch.cat([single_negative_image_embeds, single_image_embeds])117            single_image_embeds = single_image_embeds.to(device)118 119        image_embeds.append(single_image_embeds)120 121    return image_embeds122 123 124vae = AutoencoderKL.from_pretrained(125    "madebyollin/sdxl-vae-fp16-fix",126    torch_dtype=torch.float16,127).to("cuda")128 129pipeline = StableDiffusionXLPipeline.from_pretrained(130    "RunDiffusion/Juggernaut-XL-v9",131    torch_dtype=torch.float16,132    vae=vae,133    variant="fp16",134).to("cuda")135 136pipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config)137pipeline.scheduler.config.use_karras_sigmas = True138 139pipeline.load_ip_adapter(140    "h94/IP-Adapter",141    subfolder="sdxl_models",142    weight_name="ip-adapter-plus_sdxl_vit-h.safetensors",143    image_encoder_folder="models/image_encoder",144)145pipeline.set_ip_adapter_scale(0.7)146 147ip_image = load_image("source.png")148negative_ip_image = load_image("noise.png")149 150image_embeds = prepare_ip_adapter_image_embeds(151    unet=pipeline.unet,152    image_encoder=pipeline.image_encoder,153    feature_extractor=pipeline.feature_extractor,154    ip_adapter_image=[[ip_image]],155    do_classifier_free_guidance=True,156    device="cuda",157    num_images_per_prompt=1,158    ip_adapter_negative_image=negative_ip_image,159)160 161 162prompt = "cinematic photo of a cyborg in the city, 4k, high quality, intricate, highly detailed"163negative_prompt = "blurry, smooth, plastic"164 165image = pipeline(166    prompt=prompt,167    negative_prompt=negative_prompt,168    ip_adapter_image_embeds=image_embeds,169    guidance_scale=6.0,170    num_inference_steps=25,171    generator=torch.Generator(device="cpu").manual_seed(1556265306),172).images[0]173 174image.save("result.png")175```176 177### Asymmetric Tiling178Stable Diffusion is not trained to generate seamless textures. However, you can use this simple script to add tiling to your generation. This script is contributed by [alexisrolland](https://github.com/alexisrolland). See more details in the [this issue](https://github.com/huggingface/diffusers/issues/556)179 180 181|Generated|Tiled|182|---|---|183|![20240313003235_573631814](https://github.com/huggingface/diffusers/assets/5442875/eca174fb-06a4-464e-a3a7-00dbb024543e)|![wall](https://github.com/huggingface/diffusers/assets/5442875/b4aa774b-2a6a-4316-a8eb-8f30b5f4d024)|184 185 186```py187import torch188from typing import Optional189from diffusers import StableDiffusionPipeline190from diffusers.models.lora import LoRACompatibleConv191 192def seamless_tiling(pipeline, x_axis, y_axis):193    def asymmetric_conv2d_convforward(self, input: torch.Tensor, weight: torch.Tensor, bias: Optional[torch.Tensor] = None):194        self.paddingX = (self._reversed_padding_repeated_twice[0], self._reversed_padding_repeated_twice[1], 0, 0)195        self.paddingY = (0, 0, self._reversed_padding_repeated_twice[2], self._reversed_padding_repeated_twice[3])196        working = torch.nn.functional.pad(input, self.paddingX, mode=x_mode)197        working = torch.nn.functional.pad(working, self.paddingY, mode=y_mode)198        return torch.nn.functional.conv2d(working, weight, bias, self.stride, torch.nn.modules.utils._pair(0), self.dilation, self.groups)199    x_mode = 'circular' if x_axis else 'constant'200    y_mode = 'circular' if y_axis else 'constant'201    targets = [pipeline.vae, pipeline.text_encoder, pipeline.unet]202    convolution_layers = []203    for target in targets:204        for module in target.modules():205            if isinstance(module, torch.nn.Conv2d):206                convolution_layers.append(module)207    for layer in convolution_layers:208        if isinstance(layer, LoRACompatibleConv) and layer.lora_layer is None:209            layer.lora_layer = lambda * x: 0210        layer._conv_forward = asymmetric_conv2d_convforward.__get__(layer, torch.nn.Conv2d)211    return pipeline212 213pipeline = StableDiffusionPipeline.from_pretrained("stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.float16, use_safetensors=True)214pipeline.enable_model_cpu_offload()215prompt = ["texture of a red brick wall"]216seed = 123456217generator = torch.Generator(device='cuda').manual_seed(seed)218 219pipeline = seamless_tiling(pipeline=pipeline, x_axis=True, y_axis=True)220image = pipeline(221    prompt=prompt,222    width=512,223    height=512,224    num_inference_steps=20,225    guidance_scale=7,226    num_images_per_prompt=1,227    generator=generator228).images[0]229seamless_tiling(pipeline=pipeline, x_axis=False, y_axis=False)230 231torch.cuda.empty_cache()232image.save('image.png')233```234 235### Prompt Scheduling callback236 237Prompt scheduling callback allows changing prompts during a generation, like [prompt editing in A1111](https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Features#prompt-editing)238 239```python240from diffusers import StableDiffusionPipeline241from diffusers.callbacks import PipelineCallback, MultiPipelineCallbacks242from diffusers.configuration_utils import register_to_config243import torch244from typing import Any, Dict, Tuple, Union245 246 247class SDPromptSchedulingCallback(PipelineCallback):248    @register_to_config249    def __init__(250        self,251        encoded_prompt: Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]],252        cutoff_step_ratio=None,253        cutoff_step_index=None,254    ):255        super().__init__(256            cutoff_step_ratio=cutoff_step_ratio, cutoff_step_index=cutoff_step_index257        )258 259    tensor_inputs = ["prompt_embeds"]260 261    def callback_fn(262        self, pipeline, step_index, timestep, callback_kwargs263    ) -> Dict[str, Any]:264        cutoff_step_ratio = self.config.cutoff_step_ratio265        cutoff_step_index = self.config.cutoff_step_index266        if isinstance(self.config.encoded_prompt, tuple):267            prompt_embeds, negative_prompt_embeds = self.config.encoded_prompt268        else:269            prompt_embeds = self.config.encoded_prompt270 271        # Use cutoff_step_index if it's not None, otherwise use cutoff_step_ratio272        cutoff_step = (273            cutoff_step_index274            if cutoff_step_index is not None275            else int(pipeline.num_timesteps * cutoff_step_ratio)276        )277 278        if step_index == cutoff_step:279            if pipeline.do_classifier_free_guidance:280                prompt_embeds = torch.cat([negative_prompt_embeds, prompt_embeds])281            callback_kwargs[self.tensor_inputs[0]] = prompt_embeds282        return callback_kwargs283 284 285pipeline: StableDiffusionPipeline = StableDiffusionPipeline.from_pretrained(286    "stable-diffusion-v1-5/stable-diffusion-v1-5",287    torch_dtype=torch.float16,288    variant="fp16",289    use_safetensors=True,290).to("cuda")291pipeline.safety_checker = None292pipeline.requires_safety_checker = False293 294callback = MultiPipelineCallbacks(295    [296        SDPromptSchedulingCallback(297            encoded_prompt=pipeline.encode_prompt(298                prompt=f"prompt {index}",299                negative_prompt=f"negative prompt {index}",300                device=pipeline._execution_device,301                num_images_per_prompt=1,302                # pipeline.do_classifier_free_guidance can't be accessed until after pipeline is ran303                do_classifier_free_guidance=True,304            ),305            cutoff_step_index=index,306        ) for index in range(1, 20)307    ]308)309 310image = pipeline(311    prompt="prompt"312    negative_prompt="negative prompt",313    callback_on_step_end=callback,314    callback_on_step_end_tensor_inputs=["prompt_embeds"],315).images[0]316torch.cuda.empty_cache()317image.save('image.png')318```319 320```python321from diffusers import StableDiffusionXLPipeline322from diffusers.callbacks import PipelineCallback, MultiPipelineCallbacks323from diffusers.configuration_utils import register_to_config324import torch325from typing import Any, Dict, Tuple, Union326 327 328class SDXLPromptSchedulingCallback(PipelineCallback):329    @register_to_config330    def __init__(331        self,332        encoded_prompt: Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]],333        add_text_embeds: Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]],334        add_time_ids: Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]],335        cutoff_step_ratio=None,336        cutoff_step_index=None,337    ):338        super().__init__(339            cutoff_step_ratio=cutoff_step_ratio, cutoff_step_index=cutoff_step_index340        )341 342    tensor_inputs = ["prompt_embeds", "add_text_embeds", "add_time_ids"]343 344    def callback_fn(345        self, pipeline, step_index, timestep, callback_kwargs346    ) -> Dict[str, Any]:347        cutoff_step_ratio = self.config.cutoff_step_ratio348        cutoff_step_index = self.config.cutoff_step_index349        if isinstance(self.config.encoded_prompt, tuple):350            prompt_embeds, negative_prompt_embeds = self.config.encoded_prompt351        else:352            prompt_embeds = self.config.encoded_prompt353        if isinstance(self.config.add_text_embeds, tuple):354            add_text_embeds, negative_add_text_embeds = self.config.add_text_embeds355        else:356            add_text_embeds = self.config.add_text_embeds357        if isinstance(self.config.add_time_ids, tuple):358            add_time_ids, negative_add_time_ids = self.config.add_time_ids359        else:360            add_time_ids = self.config.add_time_ids361 362        # Use cutoff_step_index if it's not None, otherwise use cutoff_step_ratio363        cutoff_step = (364            cutoff_step_index365            if cutoff_step_index is not None366            else int(pipeline.num_timesteps * cutoff_step_ratio)367        )368 369        if step_index == cutoff_step:370            if pipeline.do_classifier_free_guidance:371                prompt_embeds = torch.cat([negative_prompt_embeds, prompt_embeds])372                add_text_embeds = torch.cat([negative_add_text_embeds, add_text_embeds])373                add_time_ids = torch.cat([negative_add_time_ids, add_time_ids])374            callback_kwargs[self.tensor_inputs[0]] = prompt_embeds375            callback_kwargs[self.tensor_inputs[1]] = add_text_embeds376            callback_kwargs[self.tensor_inputs[2]] = add_time_ids377        return callback_kwargs378 379 380pipeline: StableDiffusionXLPipeline = StableDiffusionXLPipeline.from_pretrained(381    "stabilityai/stable-diffusion-xl-base-1.0",382    torch_dtype=torch.float16,383    variant="fp16",384    use_safetensors=True,385).to("cuda")386 387callbacks = []388for index in range(1, 20):389    (390        prompt_embeds,391        negative_prompt_embeds,392        pooled_prompt_embeds,393        negative_pooled_prompt_embeds,394    ) = pipeline.encode_prompt(395        prompt=f"prompt {index}",396        negative_prompt=f"prompt {index}",397        device=pipeline._execution_device,398        num_images_per_prompt=1,399        # pipeline.do_classifier_free_guidance can't be accessed until after pipeline is ran400        do_classifier_free_guidance=True,401    )402    text_encoder_projection_dim = int(pooled_prompt_embeds.shape[-1])403    add_time_ids = pipeline._get_add_time_ids(404        (1024, 1024),405        (0, 0),406        (1024, 1024),407        dtype=prompt_embeds.dtype,408        text_encoder_projection_dim=text_encoder_projection_dim,409    )410    negative_add_time_ids = pipeline._get_add_time_ids(411        (1024, 1024),412        (0, 0),413        (1024, 1024),414        dtype=prompt_embeds.dtype,415        text_encoder_projection_dim=text_encoder_projection_dim,416    )417    callbacks.append(418        SDXLPromptSchedulingCallback(419            encoded_prompt=(prompt_embeds, negative_prompt_embeds),420            add_text_embeds=(pooled_prompt_embeds, negative_pooled_prompt_embeds),421            add_time_ids=(add_time_ids, negative_add_time_ids),422            cutoff_step_index=index,423        )424    )425 426 427callback = MultiPipelineCallbacks(callbacks)428 429image = pipeline(430    prompt="prompt",431    negative_prompt="negative prompt",432    callback_on_step_end=callback,433    callback_on_step_end_tensor_inputs=[434        "prompt_embeds",435        "add_text_embeds",436        "add_time_ids",437    ],438).images[0]439```440