diffusers/community-pipelines-mirror
Community Pipeline Examples For more information about community pipelines, please have a look at this issue. Community pipeline examples consist pipelines that have been added by the community. Please have a look at the following tables to get an overview of all community examples. Click on the Code Example to get a copy-and-paste ready code example that you can try out. If a community pipeline doesn't work as expected, please open an issue and ping the author on it. Please… See the full description on the dataset page: https://huggingface.co/datasets/diffusers/community-pipelines-mirror.
922k
1# Community Pipeline Examples2 3> **For more information about community pipelines, please have a look at [this issue](https://github.com/huggingface/diffusers/issues/841).**4 5**Community pipeline** examples consist pipelines that have been added by the community.6Please have a look at the following tables to get an overview of all community examples. Click on the **Code Example** to get a copy-and-paste ready code example that you can try out.7If a community pipeline doesn't work as expected, please open an issue and ping the author on it.8 9Please also check out our [Community Scripts](https://github.com/huggingface/diffusers/blob/main/examples/community/README_community_scripts.md) examples for tips and tricks that you can use with diffusers without having to run a community pipeline.10 11| Example | Description | Code Example | Colab | Author |12|:--------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------:|13|Differential Diffusion|[Differential Diffusion](https://github.com/exx8/differential-diffusion) modifies an image according to a text prompt, and according to a map that specifies the amount of change in each region.|[Differential Diffusion](#differential-diffusion)|[](https://huggingface.co/spaces/exx8/differential-diffusion) [](https://colab.research.google.com/github/exx8/differential-diffusion/blob/main/examples/SD2.ipynb)|[Eran Levin](https://github.com/exx8) and [Ohad Fried](https://www.ohadf.com/)|14| HD-Painter | [HD-Painter](https://github.com/Picsart-AI-Research/HD-Painter) enables prompt-faithfull and high resolution (up to 2k) image inpainting upon any diffusion-based image inpainting method. | [HD-Painter](#hd-painter) | [](https://huggingface.co/spaces/PAIR/HD-Painter) | [Manukyan Hayk](https://github.com/haikmanukyan) and [Sargsyan Andranik](https://github.com/AndranikSargsyan) |15| Marigold Monocular Depth Estimation | A universal monocular depth estimator, utilizing Stable Diffusion, delivering sharp predictions in the wild. (See the [project page](https://marigoldmonodepth.github.io) and [full codebase](https://github.com/prs-eth/marigold) for more details.) | [Marigold Depth Estimation](#marigold-depth-estimation) | [](https://huggingface.co/spaces/toshas/marigold) [](https://colab.research.google.com/drive/12G8reD13DdpMie5ZQlaFNo2WCGeNUH-u?usp=sharing) | [Bingxin Ke](https://github.com/markkua) and [Anton Obukhov](https://github.com/toshas) |16| LLM-grounded Diffusion (LMD+) | LMD greatly improves the prompt following ability of text-to-image generation models by introducing an LLM as a front-end prompt parser and layout planner. [Project page.](https://llm-grounded-diffusion.github.io/) [See our full codebase (also with diffusers).](https://github.com/TonyLianLong/LLM-groundedDiffusion) | [LLM-grounded Diffusion (LMD+)](#llm-grounded-diffusion) | [Huggingface Demo](https://huggingface.co/spaces/longlian/llm-grounded-diffusion) [](https://colab.research.google.com/drive/1SXzMSeAB-LJYISb2yrUOdypLz4OYWUKj) | [Long (Tony) Lian](https://tonylian.com/) |17| CLIP Guided Stable Diffusion | Doing CLIP guidance for text to image generation with Stable Diffusion | [CLIP Guided Stable Diffusion](#clip-guided-stable-diffusion) | [](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/CLIP_Guided_Stable_diffusion_with_diffusers.ipynb) | [Suraj Patil](https://github.com/patil-suraj/) |18| One Step U-Net (Dummy) | Example showcasing of how to use Community Pipelines (see <https://github.com/huggingface/diffusers/issues/841>) | [One Step U-Net](#one-step-unet) | - | [Patrick von Platen](https://github.com/patrickvonplaten/) |19| Stable Diffusion Interpolation | Interpolate the latent space of Stable Diffusion between different prompts/seeds | [Stable Diffusion Interpolation](#stable-diffusion-interpolation) | - | [Nate Raw](https://github.com/nateraw/) |20| Stable Diffusion Mega | **One** Stable Diffusion Pipeline with all functionalities of [Text2Image](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py), [Image2Image](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_img2img.py) and [Inpainting](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint.py) | [Stable Diffusion Mega](#stable-diffusion-mega) | - | [Patrick von Platen](https://github.com/patrickvonplaten/) |21| Long Prompt Weighting Stable Diffusion | **One** Stable Diffusion Pipeline without tokens length limit, and support parsing weighting in prompt. | [Long Prompt Weighting Stable Diffusion](#long-prompt-weighting-stable-diffusion) | - | [SkyTNT](https://github.com/SkyTNT) |22| Speech to Image | Using automatic-speech-recognition to transcribe text and Stable Diffusion to generate images | [Speech to Image](#speech-to-image) | - | [Mikail Duzenli](https://github.com/MikailINTech)23| Wild Card Stable Diffusion | Stable Diffusion Pipeline that supports prompts that contain wildcard terms (indicated by surrounding double underscores), with values instantiated randomly from a corresponding txt file or a dictionary of possible values | [Wildcard Stable Diffusion](#wildcard-stable-diffusion) | - | [Shyam Sudhakaran](https://github.com/shyamsn97) |24| [Composable Stable Diffusion](https://energy-based-model.github.io/Compositional-Visual-Generation-with-Composable-Diffusion-Models/) | Stable Diffusion Pipeline that supports prompts that contain "|" in prompts (as an AND condition) and weights (separated by "|" as well) to positively / negatively weight prompts. | [Composable Stable Diffusion](#composable-stable-diffusion) | - | [Mark Rich](https://github.com/MarkRich) |25| Seed Resizing Stable Diffusion | Stable Diffusion Pipeline that supports resizing an image and retaining the concepts of the 512 by 512 generation. | [Seed Resizing](#seed-resizing) | - | [Mark Rich](https://github.com/MarkRich) |26| Imagic Stable Diffusion | Stable Diffusion Pipeline that enables writing a text prompt to edit an existing image | [Imagic Stable Diffusion](#imagic-stable-diffusion) | - | [Mark Rich](https://github.com/MarkRich) |27| Multilingual Stable Diffusion | Stable Diffusion Pipeline that supports prompts in 50 different languages. | [Multilingual Stable Diffusion](#multilingual-stable-diffusion-pipeline) | - | [Juan Carlos Piñeros](https://github.com/juancopi81) |28| GlueGen Stable Diffusion | Stable Diffusion Pipeline that supports prompts in different languages using GlueGen adapter. | [GlueGen Stable Diffusion](#gluegen-stable-diffusion-pipeline) | - | [Phạm Hồng Vinh](https://github.com/rootonchair) |29| Image to Image Inpainting Stable Diffusion | Stable Diffusion Pipeline that enables the overlaying of two images and subsequent inpainting | [Image to Image Inpainting Stable Diffusion](#image-to-image-inpainting-stable-diffusion) | - | [Alex McKinney](https://github.com/vvvm23) |30| Text Based Inpainting Stable Diffusion | Stable Diffusion Inpainting Pipeline that enables passing a text prompt to generate the mask for inpainting | [Text Based Inpainting Stable Diffusion](#image-to-image-inpainting-stable-diffusion) | - | [Dhruv Karan](https://github.com/unography) |31| Bit Diffusion | Diffusion on discrete data | [Bit Diffusion](#bit-diffusion) | - | [Stuti R.](https://github.com/kingstut) |32| K-Diffusion Stable Diffusion | Run Stable Diffusion with any of [K-Diffusion's samplers](https://github.com/crowsonkb/k-diffusion/blob/master/k_diffusion/sampling.py) | [Stable Diffusion with K Diffusion](#stable-diffusion-with-k-diffusion) | - | [Patrick von Platen](https://github.com/patrickvonplaten/) |33| Checkpoint Merger Pipeline | Diffusion Pipeline that enables merging of saved model checkpoints | [Checkpoint Merger Pipeline](#checkpoint-merger-pipeline) | - | [Naga Sai Abhinay Devarinti](https://github.com/Abhinay1997/) |34| Stable Diffusion v1.1-1.4 Comparison | Run all 4 model checkpoints for Stable Diffusion and compare their results together | [Stable Diffusion Comparison](#stable-diffusion-comparisons) | - | [Suvaditya Mukherjee](https://github.com/suvadityamuk) |35| MagicMix | Diffusion Pipeline for semantic mixing of an image and a text prompt | [MagicMix](#magic-mix) | - | [Partho Das](https://github.com/daspartho) |36| Stable UnCLIP | Diffusion Pipeline for combining prior model (generate clip image embedding from text, UnCLIPPipeline `"kakaobrain/karlo-v1-alpha"`) and decoder pipeline (decode clip image embedding to image, StableDiffusionImageVariationPipeline `"lambdalabs/sd-image-variations-diffusers"` ). | [Stable UnCLIP](#stable-unclip) | - | [Ray Wang](https://wrong.wang) |37| UnCLIP Text Interpolation Pipeline | Diffusion Pipeline that allows passing two prompts and produces images while interpolating between the text-embeddings of the two prompts | [UnCLIP Text Interpolation Pipeline](#unclip-text-interpolation-pipeline) | - | [Naga Sai Abhinay Devarinti](https://github.com/Abhinay1997/) |38| UnCLIP Image Interpolation Pipeline | Diffusion Pipeline that allows passing two images/image_embeddings and produces images while interpolating between their image-embeddings | [UnCLIP Image Interpolation Pipeline](#unclip-image-interpolation-pipeline) | - | [Naga Sai Abhinay Devarinti](https://github.com/Abhinay1997/) |39| DDIM Noise Comparative Analysis Pipeline | Investigating how the diffusion models learn visual concepts from each noise level (which is a contribution of [P2 weighting (CVPR 2022)](https://arxiv.org/abs/2204.00227)) | [DDIM Noise Comparative Analysis Pipeline](#ddim-noise-comparative-analysis-pipeline) | - | [Aengus (Duc-Anh)](https://github.com/aengusng8) |40| CLIP Guided Img2Img Stable Diffusion Pipeline | Doing CLIP guidance for image to image generation with Stable Diffusion | [CLIP Guided Img2Img Stable Diffusion](#clip-guided-img2img-stable-diffusion) | - | [Nipun Jindal](https://github.com/nipunjindal/) |41| TensorRT Stable Diffusion Text to Image Pipeline | Accelerates the Stable Diffusion Text2Image Pipeline using TensorRT | [TensorRT Stable Diffusion Text to Image Pipeline](#tensorrt-text2image-stable-diffusion-pipeline) | - | [Asfiya Baig](https://github.com/asfiyab-nvidia) |42| EDICT Image Editing Pipeline | Diffusion pipeline for text-guided image editing | [EDICT Image Editing Pipeline](#edict-image-editing-pipeline) | - | [Joqsan Azocar](https://github.com/Joqsan) |43| Stable Diffusion RePaint | Stable Diffusion pipeline using [RePaint](https://arxiv.org/abs/2201.0986) for inpainting. | [Stable Diffusion RePaint](#stable-diffusion-repaint ) | - | [Markus Pobitzer](https://github.com/Markus-Pobitzer) |44| TensorRT Stable Diffusion Image to Image Pipeline | Accelerates the Stable Diffusion Image2Image Pipeline using TensorRT | [TensorRT Stable Diffusion Image to Image Pipeline](#tensorrt-image2image-stable-diffusion-pipeline) | - | [Asfiya Baig](https://github.com/asfiyab-nvidia) |45| Stable Diffusion IPEX Pipeline | Accelerate Stable Diffusion inference pipeline with BF16/FP32 precision on Intel Xeon CPUs with [IPEX](https://github.com/intel/intel-extension-for-pytorch) | [Stable Diffusion on IPEX](#stable-diffusion-on-ipex) | - | [Yingjie Han](https://github.com/yingjie-han/) |46| CLIP Guided Images Mixing Stable Diffusion Pipeline | Сombine images using usual diffusion models. | [CLIP Guided Images Mixing Using Stable Diffusion](#clip-guided-images-mixing-with-stable-diffusion) | - | [Karachev Denis](https://github.com/TheDenk) |47| TensorRT Stable Diffusion Inpainting Pipeline | Accelerates the Stable Diffusion Inpainting Pipeline using TensorRT | [TensorRT Stable Diffusion Inpainting Pipeline](#tensorrt-inpainting-stable-diffusion-pipeline) | - | [Asfiya Baig](https://github.com/asfiyab-nvidia) |48| IADB Pipeline | Implementation of [Iterative α-(de)Blending: a Minimalist Deterministic Diffusion Model](https://arxiv.org/abs/2305.03486) | [IADB Pipeline](#iadb-pipeline) | - | [Thomas Chambon](https://github.com/tchambon)49| Zero1to3 Pipeline | Implementation of [Zero-1-to-3: Zero-shot One Image to 3D Object](https://arxiv.org/abs/2303.11328) | [Zero1to3 Pipeline](#zero1to3-pipeline) | - | [Xin Kong](https://github.com/kxhit) |50| Stable Diffusion XL Long Weighted Prompt Pipeline | A pipeline support unlimited length of prompt and negative prompt, use A1111 style of prompt weighting | [Stable Diffusion XL Long Weighted Prompt Pipeline](#stable-diffusion-xl-long-weighted-prompt-pipeline) | [](https://colab.research.google.com/drive/1LsqilswLR40XLLcp6XFOl5nKb_wOe26W?usp=sharing) | [Andrew Zhu](https://xhinker.medium.com/) |51| FABRIC - Stable Diffusion with feedback Pipeline | pipeline supports feedback from liked and disliked images | [Stable Diffusion Fabric Pipeline](#stable-diffusion-fabric-pipeline) | - | [Shauray Singh](https://shauray8.github.io/about_shauray/) |52| sketch inpaint - Inpainting with non-inpaint Stable Diffusion | sketch inpaint much like in automatic1111 | [Masked Im2Im Stable Diffusion Pipeline](#stable-diffusion-masked-im2im) | - | [Anatoly Belikov](https://github.com/noskill) |53| prompt-to-prompt | change parts of a prompt and retain image structure (see [paper page](https://prompt-to-prompt.github.io/)) | [Prompt2Prompt Pipeline](#prompt2prompt-pipeline) | - | [Umer H. Adil](https://twitter.com/UmerHAdil) |54| Latent Consistency Pipeline | Implementation of [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://arxiv.org/abs/2310.04378) | [Latent Consistency Pipeline](#latent-consistency-pipeline) | - | [Simian Luo](https://github.com/luosiallen) |55| Latent Consistency Img2img Pipeline | Img2img pipeline for Latent Consistency Models | [Latent Consistency Img2Img Pipeline](#latent-consistency-img2img-pipeline) | - | [Logan Zoellner](https://github.com/nagolinc) |56| Latent Consistency Interpolation Pipeline | Interpolate the latent space of Latent Consistency Models with multiple prompts | [Latent Consistency Interpolation Pipeline](#latent-consistency-interpolation-pipeline) | [](https://colab.research.google.com/drive/1pK3NrLWJSiJsBynLns1K1-IDTW9zbPvl?usp=sharing) | [Aryan V S](https://github.com/a-r-r-o-w) |57| SDE Drag Pipeline | The pipeline supports drag editing of images using stochastic differential equations | [SDE Drag Pipeline](#sde-drag-pipeline) | - | [NieShen](https://github.com/NieShenRuc) [Fengqi Zhu](https://github.com/Monohydroxides) |58| Regional Prompting Pipeline | Assign multiple prompts for different regions | [Regional Prompting Pipeline](#regional-prompting-pipeline) | - | [hako-mikan](https://github.com/hako-mikan) |59| LDM3D-sr (LDM3D upscaler) | Upscale low resolution RGB and depth inputs to high resolution | [StableDiffusionUpscaleLDM3D Pipeline](https://github.com/estelleafl/diffusers/tree/ldm3d_upscaler_community/examples/community#stablediffusionupscaleldm3d-pipeline) | - | [Estelle Aflalo](https://github.com/estelleafl) |60| AnimateDiff ControlNet Pipeline | Combines AnimateDiff with precise motion control using ControlNets | [AnimateDiff ControlNet Pipeline](#animatediff-controlnet-pipeline) | [](https://colab.research.google.com/drive/1SKboYeGjEQmQPWoFC0aLYpBlYdHXkvAu?usp=sharing) | [Aryan V S](https://github.com/a-r-r-o-w) and [Edoardo Botta](https://github.com/EdoardoBotta) |61| DemoFusion Pipeline | Implementation of [DemoFusion: Democratising High-Resolution Image Generation With No $$$](https://arxiv.org/abs/2311.16973) | [DemoFusion Pipeline](#demofusion) | - | [Ruoyi Du](https://github.com/RuoyiDu) |62| Instaflow Pipeline | Implementation of [InstaFlow! One-Step Stable Diffusion with Rectified Flow](https://arxiv.org/abs/2309.06380) | [Instaflow Pipeline](#instaflow-pipeline) | - | [Ayush Mangal](https://github.com/ayushtues) |63| Null-Text Inversion Pipeline | Implement [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://arxiv.org/abs/2211.09794) as a pipeline. | [Null-Text Inversion](https://github.com/google/prompt-to-prompt/) | - | [Junsheng Luan](https://github.com/Junsheng121) |64| Rerender A Video Pipeline | Implementation of [[SIGGRAPH Asia 2023] Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation](https://arxiv.org/abs/2306.07954) | [Rerender A Video Pipeline](#rerender-a-video) | - | [Yifan Zhou](https://github.com/SingleZombie) |65| StyleAligned Pipeline | Implementation of [Style Aligned Image Generation via Shared Attention](https://arxiv.org/abs/2312.02133) | [StyleAligned Pipeline](#stylealigned-pipeline) | [](https://drive.google.com/file/d/15X2E0jFPTajUIjS0FzX50OaHsCbP2lQ0/view?usp=sharing) | [Aryan V S](https://github.com/a-r-r-o-w) |66| AnimateDiff Image-To-Video Pipeline | Experimental Image-To-Video support for AnimateDiff (open to improvements) | [AnimateDiff Image To Video Pipeline](#animatediff-image-to-video-pipeline) | [](https://drive.google.com/file/d/1TvzCDPHhfFtdcJZe4RLloAwyoLKuttWK/view?usp=sharing) | [Aryan V S](https://github.com/a-r-r-o-w) |67| IP Adapter FaceID Stable Diffusion | Stable Diffusion Pipeline that supports IP Adapter Face ID | [IP Adapter Face ID](#ip-adapter-face-id) | - | [Fabio Rigano](https://github.com/fabiorigano) |68| InstantID Pipeline | Stable Diffusion XL Pipeline that supports InstantID | [InstantID Pipeline](#instantid-pipeline) | [](https://huggingface.co/spaces/InstantX/InstantID) | [Haofan Wang](https://github.com/haofanwang) |69| UFOGen Scheduler | Scheduler for UFOGen Model (compatible with Stable Diffusion pipelines) | [UFOGen Scheduler](#ufogen-scheduler) | - | [dg845](https://github.com/dg845) |70| Stable Diffusion XL IPEX Pipeline | Accelerate Stable Diffusion XL inference pipeline with BF16/FP32 precision on Intel Xeon CPUs with [IPEX](https://github.com/intel/intel-extension-for-pytorch) | [Stable Diffusion XL on IPEX](#stable-diffusion-xl-on-ipex) | - | [Dan Li](https://github.com/ustcuna/) |71| Stable Diffusion BoxDiff Pipeline | Training-free controlled generation with bounding boxes using [BoxDiff](https://github.com/showlab/BoxDiff) | [Stable Diffusion BoxDiff Pipeline](#stable-diffusion-boxdiff) | - | [Jingyang Zhang](https://github.com/zjysteven/) |72| FRESCO V2V Pipeline | Implementation of [[CVPR 2024] FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation](https://arxiv.org/abs/2403.12962) | [FRESCO V2V Pipeline](#fresco) | - | [Yifan Zhou](https://github.com/SingleZombie) |73 74To load a custom pipeline you just need to pass the `custom_pipeline` argument to `DiffusionPipeline`, as one of the files in `diffusers/examples/community`. Feel free to send a PR with your own pipelines, we will merge them quickly.75 76```py77pipe = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", custom_pipeline="filename_in_the_community_folder")78```79 80## Example usages81 82### Differential Diffusion83 84**Eran Levin, Ohad Fried**85 86**Tel Aviv University, Reichman University**87 88Diffusion models have revolutionized image generation and editing, producing state-of-the-art results in conditioned and unconditioned image synthesis. While current techniques enable user control over the degree of change in an image edit, the controllability is limited to global changes over an entire edited region. This paper introduces a novel framework that enables customization of the amount of change per pixel or per image region. Our framework can be integrated into any existing diffusion model, enhancing it with this capability. Such granular control on the quantity of change opens up a diverse array of new editing capabilities, such as control of the extent to which individual objects are modified, or the ability to introduce gradual spatial changes. Furthermore, we showcase the framework's effectiveness in soft-inpainting---the completion of portions of an image while subtly adjusting the surrounding areas to ensure seamless integration. Additionally, we introduce a new tool for exploring the effects of different change quantities. Our framework operates solely during inference, requiring no model training or fine-tuning. We demonstrate our method with the current open state-of-the-art models, and validate it via both quantitative and qualitative comparisons, and a user study.89 9091 92You can find additional information about Differential Diffusion in the [paper](https://differential-diffusion.github.io/paper.pdf) or in the [project website](https://differential-diffusion.github.io/).93 94#### Usage example95 96```python97import torch98from torchvision import transforms99 100from diffusers import DPMSolverMultistepScheduler101from diffusers.utils import load_image102from examples.community.pipeline_stable_diffusion_xl_differential_img2img import (103 StableDiffusionXLDifferentialImg2ImgPipeline,104)105 106 107pipeline = StableDiffusionXLDifferentialImg2ImgPipeline.from_pretrained(108 "SG161222/RealVisXL_V4.0", torch_dtype=torch.float16, variant="fp16"109).to("cuda")110pipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config, use_karras_sigmas=True)111 112 113def preprocess_image(image):114 image = image.convert("RGB")115 image = transforms.CenterCrop((image.size[1] // 64 * 64, image.size[0] // 64 * 64))(image)116 image = transforms.ToTensor()(image)117 image = image * 2 - 1118 image = image.unsqueeze(0).to("cuda")119 return image120 121 122def preprocess_map(map):123 map = map.convert("L")124 map = transforms.CenterCrop((map.size[1] // 64 * 64, map.size[0] // 64 * 64))(map)125 map = transforms.ToTensor()(map)126 map = map.to("cuda")127 return map128 129 130image = preprocess_image(131 load_image(132 "https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/differential/20240329211129_4024911930.png?download=true"133 )134)135 136mask = preprocess_map(137 load_image(138 "https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/differential/gradient_mask.png?download=true"139 )140)141 142prompt = "a green pear"143negative_prompt = "blurry"144 145image = pipeline(146 prompt=prompt,147 negative_prompt=negative_prompt,148 guidance_scale=7.5,149 num_inference_steps=25,150 original_image=image,151 image=image,152 strength=1.0,153 map=mask,154).images[0]155 156image.save("result.png")157```158 159### HD-Painter160 161Implementation of [HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models](https://arxiv.org/abs/2312.14091).162 163164 165The abstract from the paper is:166 167Recent progress in text-guided image inpainting, based on the unprecedented success of text-to-image diffusion models, has led to exceptionally realistic and visually plausible results.168However, there is still significant potential for improvement in current text-to-image inpainting models, particularly in better aligning the inpainted area with user prompts and performing high-resolution inpainting.169Therefore, in this paper we introduce _HD-Painter_, a completely **training-free** approach that **accurately follows to prompts** and coherently **scales to high-resolution** image inpainting.170To this end, we design the _Prompt-Aware Introverted Attention (PAIntA)_ layer enhancing self-attention scores by prompt information and resulting in better text alignment generations.171To further improve the prompt coherence we introduce the _Reweighting Attention Score Guidance (RASG)_ mechanism seamlessly integrating a post-hoc sampling strategy into general form of DDIM to prevent out-of-distribution latent shifts.172Moreover, HD-Painter allows extension to larger scales by introducing a specialized super-resolution technique customized for inpainting, enabling the completion of missing regions in images of up to 2K resolution.173Our experiments demonstrate that HD-Painter surpasses existing state-of-the-art approaches qualitatively and quantitatively, achieving an impressive generation accuracy improvement of **61.4** vs **51.9**.174We will make the codes publicly available.175 176You can find additional information about Text2Video-Zero in the [paper](https://arxiv.org/abs/2312.14091) or the [original codebase](https://github.com/Picsart-AI-Research/HD-Painter).177 178#### Usage example179 180```python181import torch182from diffusers import DiffusionPipeline, DDIMScheduler183from diffusers.utils import load_image, make_image_grid184 185pipe = DiffusionPipeline.from_pretrained(186 "stabilityai/stable-diffusion-2-inpainting",187 custom_pipeline="hd_painter"188)189pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)190 191prompt = "wooden boat"192init_image = load_image("https://raw.githubusercontent.com/Picsart-AI-Research/HD-Painter/main/__assets__/samples/images/2.jpg")193mask_image = load_image("https://raw.githubusercontent.com/Picsart-AI-Research/HD-Painter/main/__assets__/samples/masks/2.png")194 195image = pipe (prompt, init_image, mask_image, use_rasg = True, use_painta = True, generator=torch.manual_seed(12345)).images[0]196 197make_image_grid([init_image, mask_image, image], rows=1, cols=3)198 199```200 201### Marigold Depth Estimation202 203Marigold is a universal monocular depth estimator that delivers accurate and sharp predictions in the wild. Based on Stable Diffusion, it is trained exclusively with synthetic depth data and excels in zero-shot adaptation to real-world imagery. This pipeline is an official implementation of the inference process. More details can be found on our [project page](https://marigoldmonodepth.github.io) and [full codebase](https://github.com/prs-eth/marigold) (also implemented with diffusers).204 205206 207This depth estimation pipeline processes a single input image through multiple diffusion denoising stages to estimate depth maps. These maps are subsequently merged to produce the final output. Below is an example code snippet, including optional arguments:208 209```python210import numpy as np211import torch212from PIL import Image213from diffusers import DiffusionPipeline214from diffusers.utils import load_image215 216# Original DDIM version (higher quality)217pipe = DiffusionPipeline.from_pretrained(218 "prs-eth/marigold-v1-0",219 custom_pipeline="marigold_depth_estimation"220 # torch_dtype=torch.float16, # (optional) Run with half-precision (16-bit float).221 # variant="fp16", # (optional) Use with `torch_dtype=torch.float16`, to directly load fp16 checkpoint222)223 224# (New) LCM version (faster speed)225pipe = DiffusionPipeline.from_pretrained(226 "prs-eth/marigold-lcm-v1-0",227 custom_pipeline="marigold_depth_estimation"228 # torch_dtype=torch.float16, # (optional) Run with half-precision (16-bit float).229 # variant="fp16", # (optional) Use with `torch_dtype=torch.float16`, to directly load fp16 checkpoint230)231 232pipe.to("cuda")233 234img_path_or_url = "https://share.phys.ethz.ch/~pf/bingkedata/marigold/pipeline_example.jpg"235image: Image.Image = load_image(img_path_or_url)236 237pipeline_output = pipe(238 image, # Input image.239 # ----- recommended setting for DDIM version -----240 # denoising_steps=10, # (optional) Number of denoising steps of each inference pass. Default: 10.241 # ensemble_size=10, # (optional) Number of inference passes in the ensemble. Default: 10.242 # ------------------------------------------------243 244 # ----- recommended setting for LCM version ------245 # denoising_steps=4,246 # ensemble_size=5,247 # -------------------------------------------------248 249 # processing_res=768, # (optional) Maximum resolution of processing. If set to 0: will not resize at all. Defaults to 768.250 # match_input_res=True, # (optional) Resize depth prediction to match input resolution.251 # batch_size=0, # (optional) Inference batch size, no bigger than `num_ensemble`. If set to 0, the script will automatically decide the proper batch size. Defaults to 0.252 # seed=2024, # (optional) Random seed can be set to ensure additional reproducibility. Default: None (unseeded). Note: forcing --batch_size 1 helps to increase reproducibility. To ensure full reproducibility, deterministic mode needs to be used.253 # color_map="Spectral", # (optional) Colormap used to colorize the depth map. Defaults to "Spectral". Set to `None` to skip colormap generation.254 # show_progress_bar=True, # (optional) If true, will show progress bars of the inference progress.255)256 257depth: np.ndarray = pipeline_output.depth_np # Predicted depth map258depth_colored: Image.Image = pipeline_output.depth_colored # Colorized prediction259 260# Save as uint16 PNG261depth_uint16 = (depth * 65535.0).astype(np.uint16)262Image.fromarray(depth_uint16).save("./depth_map.png", mode="I;16")263 264# Save colorized depth map265depth_colored.save("./depth_colored.png")266```267 268### LLM-grounded Diffusion269 270LMD and LMD+ greatly improves the prompt understanding ability of text-to-image generation models by introducing an LLM as a front-end prompt parser and layout planner. It improves spatial reasoning, the understanding of negation, attribute binding, generative numeracy, etc. in a unified manner without explicitly aiming for each. LMD is completely training-free (i.e., uses SD model off-the-shelf). LMD+ takes in additional adapters for better control. This is a reproduction of LMD+ model used in our work. [Project page.](https://llm-grounded-diffusion.github.io/) [See our full codebase (also with diffusers).](https://github.com/TonyLianLong/LLM-groundedDiffusion)271 272273274 275This pipeline can be used with an LLM or on its own. We provide a parser that parses LLM outputs to the layouts. You can obtain the prompt to input to the LLM for layout generation [here](https://github.com/TonyLianLong/LLM-groundedDiffusion/blob/main/prompt.py). After feeding the prompt to an LLM (e.g., GPT-4 on ChatGPT website), you can feed the LLM response into our pipeline.276 277The following code has been tested on 1x RTX 4090, but it should also support GPUs with lower GPU memory.278 279#### Use this pipeline with an LLM280 281```python282import torch283from diffusers import DiffusionPipeline284 285pipe = DiffusionPipeline.from_pretrained(286 "longlian/lmd_plus",287 custom_pipeline="llm_grounded_diffusion",288 custom_revision="main",289 variant="fp16", torch_dtype=torch.float16290)291pipe.enable_model_cpu_offload()292 293# Generate directly from a text prompt and an LLM response294prompt = "a waterfall and a modern high speed train in a beautiful forest with fall foliage"295phrases, boxes, bg_prompt, neg_prompt = pipe.parse_llm_response("""296[('a waterfall', [71, 105, 148, 258]), ('a modern high speed train', [255, 223, 181, 149])]297Background prompt: A beautiful forest with fall foliage298Negative prompt:299""")300 301images = pipe(302 prompt=prompt,303 negative_prompt=neg_prompt,304 phrases=phrases,305 boxes=boxes,306 gligen_scheduled_sampling_beta=0.4,307 output_type="pil",308 num_inference_steps=50,309 lmd_guidance_kwargs={}310).images311 312images[0].save("./lmd_plus_generation.jpg")313```314 315#### Use this pipeline on its own for layout generation316 317```python318import torch319from diffusers import DiffusionPipeline320 321pipe = DiffusionPipeline.from_pretrained(322 "longlian/lmd_plus",323 custom_pipeline="llm_grounded_diffusion",324 variant="fp16", torch_dtype=torch.float16325)326pipe.enable_model_cpu_offload()327 328# Generate an image described by the prompt and329# insert objects described by text at the region defined by bounding boxes330prompt = "a waterfall and a modern high speed train in a beautiful forest with fall foliage"331boxes = [[0.1387, 0.2051, 0.4277, 0.7090], [0.4980, 0.4355, 0.8516, 0.7266]]332phrases = ["a waterfall", "a modern high speed train"]333 334images = pipe(335 prompt=prompt,336 phrases=phrases,337 boxes=boxes,338 gligen_scheduled_sampling_beta=0.4,339 output_type="pil",340 num_inference_steps=50,341 lmd_guidance_kwargs={}342).images343 344images[0].save("./lmd_plus_generation.jpg")345```346 347### CLIP Guided Stable Diffusion348 349CLIP guided stable diffusion can help to generate more realistic images350by guiding stable diffusion at every denoising step with an additional CLIP model.351 352The following code requires roughly 12GB of GPU RAM.353 354```python355from diffusers import DiffusionPipeline356from transformers import CLIPImageProcessor, CLIPModel357import torch358 359 360feature_extractor = CLIPImageProcessor.from_pretrained("laion/CLIP-ViT-B-32-laion2B-s34B-b79K")361clip_model = CLIPModel.from_pretrained("laion/CLIP-ViT-B-32-laion2B-s34B-b79K", torch_dtype=torch.float16)362 363 364guided_pipeline = DiffusionPipeline.from_pretrained(365 "runwayml/stable-diffusion-v1-5",366 custom_pipeline="clip_guided_stable_diffusion",367 clip_model=clip_model,368 feature_extractor=feature_extractor,369 370 torch_dtype=torch.float16,371)372guided_pipeline.enable_attention_slicing()373guided_pipeline = guided_pipeline.to("cuda")374 375prompt = "fantasy book cover, full moon, fantasy forest landscape, golden vector elements, fantasy magic, dark light night, intricate, elegant, sharp focus, illustration, highly detailed, digital painting, concept art, matte, art by WLOP and Artgerm and Albert Bierstadt, masterpiece"376 377generator = torch.Generator(device="cuda").manual_seed(0)378images = []379for i in range(4):380 image = guided_pipeline(381 prompt,382 num_inference_steps=50,383 guidance_scale=7.5,384 clip_guidance_scale=100,385 num_cutouts=4,386 use_cutouts=False,387 generator=generator,388 ).images[0]389 images.append(image)390 391# save images locally392for i, img in enumerate(images):393 img.save(f"./clip_guided_sd/image_{i}.png")394```395 396The `images` list contains a list of PIL images that can be saved locally or displayed directly in a google colab.397Generated images tend to be of higher qualtiy than natively using stable diffusion. E.g. the above script generates the following images:398 399.400 401### One Step Unet402 403The dummy "one-step-unet" can be run as follows:404 405```python406from diffusers import DiffusionPipeline407 408pipe = DiffusionPipeline.from_pretrained("google/ddpm-cifar10-32", custom_pipeline="one_step_unet")409pipe()410```411 412**Note**: This community pipeline is not useful as a feature, but rather just serves as an example of how community pipelines can be added (see <https://github.com/huggingface/diffusers/issues/841>).413 414### Stable Diffusion Interpolation415 416The following code can be run on a GPU of at least 8GB VRAM and should take approximately 5 minutes.417 418```python419from diffusers import DiffusionPipeline420import torch421 422pipe = DiffusionPipeline.from_pretrained(423 "CompVis/stable-diffusion-v1-4",424 revision='fp16',425 torch_dtype=torch.float16,426 safety_checker=None, # Very important for videos...lots of false positives while interpolating427 custom_pipeline="interpolate_stable_diffusion",428).to('cuda')429pipe.enable_attention_slicing()430 431frame_filepaths = pipe.walk(432 prompts=['a dog', 'a cat', 'a horse'],433 seeds=[42, 1337, 1234],434 num_interpolation_steps=16,435 output_dir='./dreams',436 batch_size=4,437 height=512,438 width=512,439 guidance_scale=8.5,440 num_inference_steps=50,441)442```443 444The output of the `walk(...)` function returns a list of images saved under the folder as defined in `output_dir`. You can use these images to create videos of stable diffusion.445 446> **Please have a look at <https://github.com/nateraw/stable-diffusion-videos> for more in-detail information on how to create videos using stable diffusion as well as more feature-complete functionality.**447 448### Stable Diffusion Mega449 450The Stable Diffusion Mega Pipeline lets you use the main use cases of the stable diffusion pipeline in a single class.451 452```python453#!/usr/bin/env python3454from diffusers import DiffusionPipeline455import PIL456import requests457from io import BytesIO458import torch459 460 461def download_image(url):462 response = requests.get(url)463 return PIL.Image.open(BytesIO(response.content)).convert("RGB")464 465pipe = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", custom_pipeline="stable_diffusion_mega", torch_dtype=torch.float16, revision="fp16")466pipe.to("cuda")467pipe.enable_attention_slicing()468 469 470### Text-to-Image471 472images = pipe.text2img("An astronaut riding a horse").images473 474### Image-to-Image475 476init_image = download_image("https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg")477 478prompt = "A fantasy landscape, trending on artstation"479 480images = pipe.img2img(prompt=prompt, image=init_image, strength=0.75, guidance_scale=7.5).images481 482### Inpainting483 484img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"485mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"486init_image = download_image(img_url).resize((512, 512))487mask_image = download_image(mask_url).resize((512, 512))488 489prompt = "a cat sitting on a bench"490images = pipe.inpaint(prompt=prompt, image=init_image, mask_image=mask_image, strength=0.75).images491```492 493As shown above this one pipeline can run all both "text-to-image", "image-to-image", and "inpainting" in one pipeline.494 495### Long Prompt Weighting Stable Diffusion496 497Features of this custom pipeline:498 499- Input a prompt without the 77 token length limit.500- Includes tx2img, img2img. and inpainting pipelines.501- Emphasize/weigh part of your prompt with parentheses as so: `a baby deer with (big eyes)`502- De-emphasize part of your prompt as so: `a [baby] deer with big eyes`503- Precisely weigh part of your prompt as so: `a baby deer with (big eyes:1.3)`504 505Prompt weighting equivalents:506 507- `a baby deer with` == `(a baby deer with:1.0)`508- `(big eyes)` == `(big eyes:1.1)`509- `((big eyes))` == `(big eyes:1.21)`510- `[big eyes]` == `(big eyes:0.91)`511 512You can run this custom pipeline as so:513 514#### pytorch515 516```python517from diffusers import DiffusionPipeline518import torch519 520pipe = DiffusionPipeline.from_pretrained(521 'hakurei/waifu-diffusion',522 custom_pipeline="lpw_stable_diffusion",523 524 torch_dtype=torch.float16525)526pipe=pipe.to("cuda")527 528prompt = "best_quality (1girl:1.3) bow bride brown_hair closed_mouth frilled_bow frilled_hair_tubes frills (full_body:1.3) fox_ear hair_bow hair_tubes happy hood japanese_clothes kimono long_sleeves red_bow smile solo tabi uchikake white_kimono wide_sleeves cherry_blossoms"529neg_prompt = "lowres, bad_anatomy, error_body, error_hair, error_arm, error_hands, bad_hands, error_fingers, bad_fingers, missing_fingers, error_legs, bad_legs, multiple_legs, missing_legs, error_lighting, error_shadow, error_reflection, text, error, extra_digit, fewer_digits, cropped, worst_quality, low_quality, normal_quality, jpeg_artifacts, signature, watermark, username, blurry"530 531pipe.text2img(prompt, negative_prompt=neg_prompt, width=512,height=512,max_embeddings_multiples=3).images[0]532 533```534 535#### onnxruntime536 537```python538from diffusers import DiffusionPipeline539import torch540 541pipe = DiffusionPipeline.from_pretrained(542 'CompVis/stable-diffusion-v1-4',543 custom_pipeline="lpw_stable_diffusion_onnx",544 revision="onnx",545 provider="CUDAExecutionProvider"546)547 548prompt = "a photo of an astronaut riding a horse on mars, best quality"549neg_prompt = "lowres, bad anatomy, error body, error hair, error arm, error hands, bad hands, error fingers, bad fingers, missing fingers, error legs, bad legs, multiple legs, missing legs, error lighting, error shadow, error reflection, text, error, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry"550 551pipe.text2img(prompt,negative_prompt=neg_prompt, width=512, height=512, max_embeddings_multiples=3).images[0]552 553```554 555if you see `Token indices sequence length is longer than the specified maximum sequence length for this model ( *** > 77 ) . Running this sequence through the model will result in indexing errors`. Do not worry, it is normal.556 557### Speech to Image558 559The following code can generate an image from an audio sample using pre-trained OpenAI whisper-small and Stable Diffusion.560 561```Python562import torch563 564import matplotlib.pyplot as plt565from datasets import load_dataset566from diffusers import DiffusionPipeline567from transformers import (568 WhisperForConditionalGeneration,569 WhisperProcessor,570)571 572 573device = "cuda" if torch.cuda.is_available() else "cpu"574 575ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")576 577audio_sample = ds[3]578 579text = audio_sample["text"].lower()580speech_data = audio_sample["audio"]["array"]581 582model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small").to(device)583processor = WhisperProcessor.from_pretrained("openai/whisper-small")584 585diffuser_pipeline = DiffusionPipeline.from_pretrained(586 "CompVis/stable-diffusion-v1-4",587 custom_pipeline="speech_to_image_diffusion",588 speech_model=model,589 speech_processor=processor,590 591 torch_dtype=torch.float16,592)593 594diffuser_pipeline.enable_attention_slicing()595diffuser_pipeline = diffuser_pipeline.to(device)596 597output = diffuser_pipeline(speech_data)598plt.imshow(output.images[0])599```600 601This example produces the following image:602 603604 605### Wildcard Stable Diffusion606 607Following the great examples from <https://github.com/jtkelm2/stable-diffusion-webui-1/blob/master/scripts/wildcards.py> and <https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Custom-Scripts#wildcards>, here's a minimal implementation that allows for users to add "wildcards", denoted by `__wildcard__` to prompts that are used as placeholders for randomly sampled values given by either a dictionary or a `.txt` file. For example:608 609Say we have a prompt:610 611```612prompt = "__animal__ sitting on a __object__ wearing a __clothing__"613```614 615We can then define possible values to be sampled for `animal`, `object`, and `clothing`. These can either be from a `.txt` with the same name as the category.616 617The possible values can also be defined / combined by using a dictionary like: `{"animal":["dog", "cat", mouse"]}`.618 619The actual pipeline works just like `StableDiffusionPipeline`, except the `__call__` method takes in:620 621`wildcard_files`: list of file paths for wild card replacement622`wildcard_option_dict`: dict with key as `wildcard` and values as a list of possible replacements623`num_prompt_samples`: number of prompts to sample, uniformly sampling wildcards624 625A full example:626 627create `animal.txt`, with contents like:628 629```630dog631cat632mouse633```634 635create `object.txt`, with contents like:636 637```638chair639sofa640bench641```642 643```python644from diffusers import DiffusionPipeline645import torch646 647pipe = DiffusionPipeline.from_pretrained(648 "CompVis/stable-diffusion-v1-4",649 custom_pipeline="wildcard_stable_diffusion",650 651 torch_dtype=torch.float16,652)653prompt = "__animal__ sitting on a __object__ wearing a __clothing__"654out = pipe(655 prompt,656 wildcard_option_dict={657 "clothing":["hat", "shirt", "scarf", "beret"]658 },659 wildcard_files=["object.txt", "animal.txt"],660 num_prompt_samples=1661)662```663 664### Composable Stable diffusion665 666[Composable Stable Diffusion](https://energy-based-model.github.io/Compositional-Visual-Generation-with-Composable-Diffusion-Models/) proposes conjunction and negation (negative prompts) operators for compositional generation with conditional diffusion models.667 668```python669import torch as th670import numpy as np671import torchvision.utils as tvu672 673from diffusers import DiffusionPipeline674 675import argparse676 677parser = argparse.ArgumentParser()678parser.add_argument("--prompt", type=str, default="mystical trees | A magical pond | dark",679 help="use '|' as the delimiter to compose separate sentences.")680parser.add_argument("--steps", type=int, default=50)681parser.add_argument("--scale", type=float, default=7.5)682parser.add_argument("--weights", type=str, default="7.5 | 7.5 | -7.5")683parser.add_argument("--seed", type=int, default=2)684parser.add_argument("--model_path", type=str, default="CompVis/stable-diffusion-v1-4")685parser.add_argument("--num_images", type=int, default=1)686args = parser.parse_args()687 688has_cuda = th.cuda.is_available()689device = th.device('cpu' if not has_cuda else 'cuda')690 691prompt = args.prompt692scale = args.scale693steps = args.steps694 695pipe = DiffusionPipeline.from_pretrained(696 args.model_path,697 custom_pipeline="composable_stable_diffusion",698).to(device)699 700pipe.safety_checker = None701 702images = []703generator = th.Generator("cuda").manual_seed(args.seed)704for i in range(args.num_images):705 image = pipe(prompt, guidance_scale=scale, num_inference_steps=steps,706 weights=args.weights, generator=generator).images[0]707 images.append(th.from_numpy(np.array(image)).permute(2, 0, 1) / 255.)708grid = tvu.make_grid(th.stack(images, dim=0), nrow=4, padding=0)709tvu.save_image(grid, f'{prompt}_{args.weights}' + '.png')710 711```712 713### Imagic Stable Diffusion714 715Allows you to edit an image using stable diffusion.716 717```python718import requests719from PIL import Image720from io import BytesIO721import torch722import os723from diffusers import DiffusionPipeline, DDIMScheduler724has_cuda = torch.cuda.is_available()725device = torch.device('cpu' if not has_cuda else 'cuda')726pipe = DiffusionPipeline.from_pretrained(727 "CompVis/stable-diffusion-v1-4",728 safety_checker=None,729 custom_pipeline="imagic_stable_diffusion",730 scheduler = DDIMScheduler(beta_start=0.00085, beta_end=0.012, beta_schedule="scaled_linear", clip_sample=False, set_alpha_to_one=False)731).to(device)732generator = torch.Generator("cuda").manual_seed(0)733seed = 0734prompt = "A photo of Barack Obama smiling with a big grin"735url = 'https://www.dropbox.com/s/6tlwzr73jd1r9yk/obama.png?dl=1'736response = requests.get(url)737init_image = Image.open(BytesIO(response.content)).convert("RGB")738init_image = init_image.resize((512, 512))739res = pipe.train(740 prompt,741 image=init_image,742 generator=generator)743res = pipe(alpha=1, guidance_scale=7.5, num_inference_steps=50)744os.makedirs("imagic", exist_ok=True)745image = res.images[0]746image.save('./imagic/imagic_image_alpha_1.png')747res = pipe(alpha=1.5, guidance_scale=7.5, num_inference_steps=50)748image = res.images[0]749image.save('./imagic/imagic_image_alpha_1_5.png')750res = pipe(alpha=2, guidance_scale=7.5, num_inference_steps=50)751image = res.images[0]752image.save('./imagic/imagic_image_alpha_2.png')753```754 755### Seed Resizing756 757Test seed resizing. Originally generate an image in 512 by 512, then generate image with same seed at 512 by 592 using seed resizing. Finally, generate 512 by 592 using original stable diffusion pipeline.758 759```python760import torch as th761import numpy as np762from diffusers import DiffusionPipeline763 764has_cuda = th.cuda.is_available()765device = th.device('cpu' if not has_cuda else 'cuda')766 767pipe = DiffusionPipeline.from_pretrained(768 "CompVis/stable-diffusion-v1-4",769 custom_pipeline="seed_resize_stable_diffusion"770).to(device)771 772def dummy(images, **kwargs):773 return images, False774 775pipe.safety_checker = dummy776 777 778images = []779th.manual_seed(0)780generator = th.Generator("cuda").manual_seed(0)781 782seed = 0783prompt = "A painting of a futuristic cop"784 785width = 512786height = 512787 788res = pipe(789 prompt,790 guidance_scale=7.5,791 num_inference_steps=50,792 height=height,793 width=width,794 generator=generator)795image = res.images[0]796image.save('./seed_resize/seed_resize_{w}_{h}_image.png'.format(w=width, h=height))797 798 799th.manual_seed(0)800generator = th.Generator("cuda").manual_seed(0)801 802pipe = DiffusionPipeline.from_pretrained(803 "CompVis/stable-diffusion-v1-4",804 custom_pipeline="/home/mark/open_source/diffusers/examples/community/"805).to(device)806 807width = 512808height = 592809 810res = pipe(811 prompt,812 guidance_scale=7.5,813 num_inference_steps=50,814 height=height,815 width=width,816 generator=generator)817image = res.images[0]818image.save('./seed_resize/seed_resize_{w}_{h}_image.png'.format(w=width, h=height))819 820pipe_compare = DiffusionPipeline.from_pretrained(821 "CompVis/stable-diffusion-v1-4",822 custom_pipeline="/home/mark/open_source/diffusers/examples/community/"823).to(device)824 825res = pipe_compare(826 prompt,827 guidance_scale=7.5,828 num_inference_steps=50,829 height=height,830 width=width,831 generator=generator832)833 834image = res.images[0]835image.save('./seed_resize/seed_resize_{w}_{h}_image_compare.png'.format(w=width, h=height))836```837 838### Multilingual Stable Diffusion Pipeline839 840The following code can generate an images from texts in different languages using the pre-trained [mBART-50 many-to-one multilingual machine translation model](https://huggingface.co/facebook/mbart-large-50-many-to-one-mmt) and Stable Diffusion.841 842```python843from PIL import Image844 845import torch846 847from diffusers import DiffusionPipeline848from transformers import (849 pipeline,850 MBart50TokenizerFast,851 MBartForConditionalGeneration,852)853device = "cuda" if torch.cuda.is_available() else "cpu"854device_dict = {"cuda": 0, "cpu": -1}855 856# helper function taken from: https://huggingface.co/blog/stable_diffusion857def image_grid(imgs, rows, cols):858 assert len(imgs) == rows*cols859 860 w, h = imgs[0].size861 grid = Image.new('RGB', size=(cols*w, rows*h))862 grid_w, grid_h = grid.size863 864 for i, img in enumerate(imgs):865 grid.paste(img, box=(i%cols*w, i//cols*h))866 return grid867 868# Add language detection pipeline869language_detection_model_ckpt = "papluca/xlm-roberta-base-language-detection"870language_detection_pipeline = pipeline("text-classification",871 model=language_detection_model_ckpt,872 device=device_dict[device])873 874# Add model for language translation875trans_tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50-many-to-one-mmt")876trans_model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-50-many-to-one-mmt").to(device)877 878diffuser_pipeline = DiffusionPipeline.from_pretrained(879 "CompVis/stable-diffusion-v1-4",880 custom_pipeline="multilingual_stable_diffusion",881 detection_pipeline=language_detection_pipeline,882 translation_model=trans_model,883 translation_tokenizer=trans_tokenizer,884 885 torch_dtype=torch.float16,886)887 888diffuser_pipeline.enable_attention_slicing()889diffuser_pipeline = diffuser_pipeline.to(device)890 891prompt = ["a photograph of an astronaut riding a horse",892 "Una casa en la playa",893 "Ein Hund, der Orange isst",894 "Un restaurant parisien"]895 896output = diffuser_pipeline(prompt)897 898images = output.images899 900grid = image_grid(images, rows=2, cols=2)901```902 903This example produces the following images:904905 906### GlueGen Stable Diffusion Pipeline907 908GlueGen is a minimal adapter that allow alignment between any encoder (Text Encoder of different language, Multilingual Roberta, AudioClip) and CLIP text encoder used in standard Stable Diffusion model. This method allows easy language adaptation to available english Stable Diffusion checkpoints without the need of an image captioning dataset as well as long training hours.909 910Make sure you downloaded `gluenet_French_clip_overnorm_over3_noln.ckpt` for French (there are also pre-trained weights for Chinese, Italian, Japanese, Spanish or train your own) at [GlueGen's official repo](https://github.com/salesforce/GlueGen/tree/main)911 912```python913from PIL import Image914 915import torch916 917from transformers import AutoModel, AutoTokenizer918 919from diffusers import DiffusionPipeline920 921if __name__ == "__main__":922 device = "cuda"923 924 lm_model_id = "xlm-roberta-large"925 token_max_length = 77926 927 text_encoder = AutoModel.from_pretrained(lm_model_id)928 tokenizer = AutoTokenizer.from_pretrained(lm_model_id, model_max_length=token_max_length, use_fast=False)929 930 tensor_norm = torch.Tensor([[43.8203],[28.3668],[27.9345],[28.0084],[28.2958],[28.2576],[28.3373],[28.2695],[28.4097],[28.2790],[28.2825],[28.2807],[28.2775],[28.2708],[28.2682],[28.2624],[28.2589],[28.2611],[28.2616],[28.2639],[28.2613],[28.2566],[28.2615],[28.2665],[28.2799],[28.2885],[28.2852],[28.2863],[28.2780],[28.2818],[28.2764],[28.2532],[28.2412],[28.2336],[28.2514],[28.2734],[28.2763],[28.2977],[28.2971],[28.2948],[28.2818],[28.2676],[28.2831],[28.2890],[28.2979],[28.2999],[28.3117],[28.3363],[28.3554],[28.3626],[28.3589],[28.3597],[28.3543],[28.3660],[28.3731],[28.3717],[28.3812],[28.3753],[28.3810],[28.3777],[28.3693],[28.3713],[28.3670],[28.3691],[28.3679],[28.3624],[28.3703],[28.3703],[28.3720],[28.3594],[28.3576],[28.3562],[28.3438],[28.3376],[28.3389],[28.3433],[28.3191]])931 932 pipeline = DiffusionPipeline.from_pretrained(933 "runwayml/stable-diffusion-v1-5",934 text_encoder=text_encoder,935 tokenizer=tokenizer,936 custom_pipeline="gluegen"937 ).to(device)938 pipeline.load_language_adapter("gluenet_French_clip_overnorm_over3_noln.ckpt", num_token=token_max_length, dim=1024, dim_out=768, tensor_norm=tensor_norm)939 940 prompt = "une voiture sur la plage"941 942 generator = torch.Generator(device=device).manual_seed(42)943 image = pipeline(prompt, generator=generator).images[0]944 image.save("gluegen_output_fr.png")945```946 947Which will produce:948 949950 951### Image to Image Inpainting Stable Diffusion952 953Similar to the standard stable diffusion inpainting example, except with the addition of an `inner_image` argument.954 955`image`, `inner_image`, and `mask` should have the same dimensions. `inner_image` should have an alpha (transparency) channel.956 957The aim is to overlay two images, then mask out the boundary between `image` and `inner_image` to allow stable diffusion to make the connection more seamless.958For example, this could be used to place a logo on a shirt and make it blend seamlessly.959 960```python961import PIL962import torch963 964from diffusers import DiffusionPipeline965 966image_path = "./path-to-image.png"967inner_image_path = "./path-to-inner-image.png"968mask_path = "./path-to-mask.png"969 970init_image = PIL.Image.open(image_path).convert("RGB").resize((512, 512))971inner_image = PIL.Image.open(inner_image_path).convert("RGBA").resize((512, 512))972mask_image = PIL.Image.open(mask_path).convert("RGB").resize((512, 512))973 974pipe = DiffusionPipeline.from_pretrained(975 "runwayml/stable-diffusion-inpainting",976 custom_pipeline="img2img_inpainting",977 978 torch_dtype=torch.float16979)980pipe = pipe.to("cuda")981 982prompt = "Your prompt here!"983image = pipe(prompt=prompt, image=init_image, inner_image=inner_image, mask_image=mask_image).images[0]984```985 986987 988### Text Based Inpainting Stable Diffusion989 990Use a text prompt to generate the mask for the area to be inpainted.991Currently uses the CLIPSeg model for mask generation, then calls the standard Stable Diffusion Inpainting pipeline to perform the inpainting.992 993```python994from transformers import CLIPSegProcessor, CLIPSegForImageSegmentation995from diffusers import DiffusionPipeline996 997from PIL import Image998import requests999 1000processor = CLIPSegProcessor.from_pretrained("CIDAS/clipseg-rd64-refined")1001model = CLIPSegForImageSegmentation.from_pretrained("CIDAS/clipseg-rd64-refined")1002 1003pipe = DiffusionPipeline.from_pretrained(1004 "runwayml/stable-diffusion-inpainting",1005 custom_pipeline="text_inpainting",1006 segmentation_model=model,1007 segmentation_processor=processor1008)1009pipe = pipe.to("cuda")1010 1011 1012url = "https://github.com/timojl/clipseg/blob/master/example_image.jpg?raw=true"1013image = Image.open(requests.get(url, stream=True).raw).resize((512, 512))1014text = "a glass" # will mask out this text1015prompt = "a cup" # the masked out region will be replaced with this1016 1017image = pipe(image=image, text=text, prompt=prompt).images[0]1018```1019 1020### Bit Diffusion1021 1022Based <https://arxiv.org/abs/2208.04202>, this is used for diffusion on discrete data - eg, discreate image data, DNA sequence data. An unconditional discreate image can be generated like this:1023 1024```python1025from diffusers import DiffusionPipeline1026pipe = DiffusionPipeline.from_pretrained("google/ddpm-cifar10-32", custom_pipeline="bit_diffusion")1027image = pipe().images[0]1028 1029```1030 1031### Stable Diffusion with K Diffusion1032 1033Make sure you have @crowsonkb's <https://github.com/crowsonkb/k-diffusion> installed:1034 1035```sh1036pip install k-diffusion1037```1038 1039You can use the community pipeline as follows:1040 1041```python1042from diffusers import DiffusionPipeline1043 1044pipe = DiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", custom_pipeline="sd_text2img_k_diffusion")1045pipe = pipe.to("cuda")1046 1047prompt = "an astronaut riding a horse on mars"1048pipe.set_scheduler("sample_heun")1049generator = torch.Generator(device="cuda").manual_seed(seed)1050image = pipe(prompt, generator=generator, num_inference_steps=20).images[0]1051 1052image.save("./astronaut_heun_k_diffusion.png")1053```1054 1055To make sure that K Diffusion and `diffusers` yield the same results:1056 1057**Diffusers**:1058 1059```python1060from diffusers import DiffusionPipeline, EulerDiscreteScheduler1061 1062seed = 331063 1064pipe = DiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4")1065pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config)1066pipe = pipe.to("cuda")1067 1068generator = torch.Generator(device="cuda").manual_seed(seed)1069image = pipe(prompt, generator=generator, num_inference_steps=50).images[0]1070```1071 10721073 1074**K Diffusion**:1075 1076```python1077from diffusers import DiffusionPipeline, EulerDiscreteScheduler1078 1079seed = 331080 1081pipe = DiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", custom_pipeline="sd_text2img_k_diffusion")1082pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config)1083pipe = pipe.to("cuda")1084 1085pipe.set_scheduler("sample_euler")1086generator = torch.Generator(device="cuda").manual_seed(seed)1087image = pipe(prompt, generator=generator, num_inference_steps=50).images[0]1088```1089 10901091 1092### Checkpoint Merger Pipeline1093 1094Based on the AUTOMATIC1111/webui for checkpoint merging. This is a custom pipeline that merges upto 3 pretrained model checkpoints as long as they are in the HuggingFace model_index.json format.1095 1096The checkpoint merging is currently memory intensive as it modifies the weights of a DiffusionPipeline object in place. Expect at least 13GB RAM Usage on Kaggle GPU kernels and1097on colab you might run out of the 12GB memory even while merging two checkpoints.1098 1099Usage:-1100 1101```python1102from diffusers import DiffusionPipeline1103 1104#Return a CheckpointMergerPipeline class that allows you to merge checkpoints.1105#The checkpoint passed here is ignored. But still pass one of the checkpoints you plan to1106#merge for convenience1107pipe = DiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", custom_pipeline="checkpoint_merger")1108 1109#There are multiple possible scenarios:1110#The pipeline with the merged checkpoints is returned in all the scenarios1111 1112#Compatible checkpoints a.k.a matched model_index.json files. Ignores the meta attributes in model_index.json during comparison.( attrs with _ as prefix )1113merged_pipe = pipe.merge(["CompVis/stable-diffusion-v1-4","CompVis/stable-diffusion-v1-2"], interp = "sigmoid", alpha = 0.4)1114 1115#Incompatible checkpoints in model_index.json but merge might be possible. Use force = True to ignore model_index.json compatibility1116merged_pipe_1 = pipe.merge(["CompVis/stable-diffusion-v1-4","hakurei/waifu-diffusion"], force = True, interp = "sigmoid", alpha = 0.4)1117 1118#Three checkpoint merging. Only "add_difference" method actually works on all three checkpoints. Using any other options will ignore the 3rd checkpoint.1119merged_pipe_2 = pipe.merge(["CompVis/stable-diffusion-v1-4","hakurei/waifu-diffusion","prompthero/openjourney"], force = True, interp = "add_difference", alpha = 0.4)1120 1121prompt = "An astronaut riding a horse on Mars"1122 1123image = merged_pipe(prompt).images[0]1124 1125```1126 1127Some examples along with the merge details:1128 11291. "CompVis/stable-diffusion-v1-4" + "hakurei/waifu-diffusion" ; Sigmoid interpolation; alpha = 0.81130 11311132 11332. "hakurei/waifu-diffusion" + "prompthero/openjourney" ; Inverse Sigmoid interpolation; alpha = 0.81134 11351136 11373. "CompVis/stable-diffusion-v1-4" + "hakurei/waifu-diffusion" + "prompthero/openjourney"; Add Difference interpolation; alpha = 0.51138 11391140 1141### Stable Diffusion Comparisons1142 1143This Community Pipeline enables the comparison between the 4 checkpoints that exist for Stable Diffusion. They can be found through the following links:1144 11451. [Stable Diffusion v1.1](https://huggingface.co/CompVis/stable-diffusion-v1-1)11462. [Stable Diffusion v1.2](https://huggingface.co/CompVis/stable-diffusion-v1-2)11473. [Stable Diffusion v1.3](https://huggingface.co/CompVis/stable-diffusion-v1-3)11484. [Stable Diffusion v1.4](https://huggingface.co/CompVis/stable-diffusion-v1-4)1149 1150```python1151from diffusers import DiffusionPipeline1152import matplotlib.pyplot as plt1153 1154pipe = DiffusionPipeline.from_pretrained('CompVis/stable-diffusion-v1-4', custom_pipeline='suvadityamuk/StableDiffusionComparison')1155pipe.enable_attention_slicing()1156pipe = pipe.to('cuda')1157prompt = "an astronaut riding a horse on mars"1158output = pipe(prompt)1159 1160plt.subplots(2,2,1)1161plt.imshow(output.images[0])1162plt.title('Stable Diffusion v1.1')1163plt.axis('off')1164plt.subplots(2,2,2)1165plt.imshow(output.images[1])1166plt.title('Stable Diffusion v1.2')1167plt.axis('off')1168plt.subplots(2,2,3)1169plt.imshow(output.images[2])1170plt.title('Stable Diffusion v1.3')1171plt.axis('off')1172plt.subplots(2,2,4)1173plt.imshow(output.images[3])1174plt.title('Stable Diffusion v1.4')1175plt.axis('off')1176 1177plt.show()1178```1179 1180As a result, you can look at a grid of all 4 generated images being shown together, that captures a difference the advancement of the training between the 4 checkpoints.1181 1182### Magic Mix1183 1184Implementation of the [MagicMix: Semantic Mixing with Diffusion Models](https://arxiv.org/abs/2210.16056) paper. This is a Diffusion Pipeline for semantic mixing of an image and a text prompt to create a new concept while preserving the spatial layout and geometry of the subject in the image. The pipeline takes an image that provides the layout semantics and a prompt that provides the content semantics for the mixing process.1185 1186There are 3 parameters for the method-1187 1188- `mix_factor`: It is the interpolation constant used in the layout generation phase. The greater the value of `mix_factor`, the greater the influence of the prompt on the layout generation process.1189- `kmax` and `kmin`: These determine the range for the layout and content generation process. A higher value of kmax results in loss of more information about the layout of the original image and a higher value of kmin results in more steps for content generation process.1190 1191Here is an example usage-1192 1193```python1194from diffusers import DiffusionPipeline, DDIMScheduler1195from PIL import Image1196 1197pipe = DiffusionPipeline.from_pretrained(1198 "CompVis/stable-diffusion-v1-4",1199 custom_pipeline="magic_mix",1200 scheduler = DDIMScheduler.from_pretrained("CompVis/stable-diffusion-v1-4", subfolder="scheduler"),