fluxdev/stable-diffusion-webui-forge
1
1---2title: stable-diffusion-webui-forge3app_file: webui.py4sdk: gradio5sdk_version: 3.41.26---7# Stable Diffusion WebUI Forge8 9Stable Diffusion WebUI Forge is a platform on top of [Stable Diffusion WebUI](https://github.com/AUTOMATIC1111/stable-diffusion-webui) (based on [Gradio](https://www.gradio.app/)) to make development easier, optimize resource management, and speed up inference.10 11The name "Forge" is inspired from "Minecraft Forge". This project is aimed at becoming SD WebUI's Forge.12 13Compared to original WebUI (for SDXL inference at 1024px), you can expect the below speed-ups:14 151. If you use common GPU like 8GB vram, you can expect to get about **30~45% speed up** in inference speed (it/s), the GPU memory peak (in task manager) will drop about 700MB to 1.3GB, the maximum diffusion resolution (that will not OOM) will increase about 2x to 3x, and the maximum diffusion batch size (that will not OOM) will increase about 4x to 6x.16 172. If you use less powerful GPU like 6GB vram, you can expect to get about **60~75% speed up** in inference speed (it/s), the GPU memory peak (in task manager) will drop about 800MB to 1.5GB, the maximum diffusion resolution (that will not OOM) will increase about 3x, and the maximum diffusion batch size (that will not OOM) will increase about 4x.18 193. If you use powerful GPU like 4090 with 24GB vram, you can expect to get about **3~6% speed up** in inference speed (it/s), the GPU memory peak (in task manager) will drop about 1GB to 1.4GB, the maximum diffusion resolution (that will not OOM) will increase about 1.6x, and the maximum diffusion batch size (that will not OOM) will increase about 2x.20 214. If you use ControlNet for SDXL, the maximum ControlNet count (that will not OOM) will increase about 2x, the speed with SDXL+ControlNet will **speed up about 30~45%**.22 23Another very important change that Forge brings is **Unet Patcher**. Using Unet Patcher, methods like Self-Attention Guidance, Kohya High Res Fix, FreeU, StyleAlign, Hypertile can all be implemented in about 100 lines of codes. 24 25Thanks to Unet Patcher, many new things are possible now and supported in Forge, including SVD, Z123, masked Ip-adapter, masked controlnet, photomaker, etc.26 27**No need to monkeypatch UNet and conflict other extensions anymore!**28 29Forge also adds a few samplers, including but not limited to DDPM, DDPM Karras, DPM++ 2M Turbo, DPM++ 2M SDE Turbo, LCM Karras, Euler A Turbo, etc. (LCM is already in original webui since 1.7.0).30 31Finally, Forge promise that we will only do our jobs. Forge will never add unnecessary opinioned changes to the user interface. You are still using 100% Automatic1111 WebUI.32 33# Installing Forge34 35If you are proficient in Git and you want to install Forge as another branch of SD-WebUI, please see [here](https://github.com/continue-revolution/sd-webui-animatediff/blob/forge/master/docs/how-to-use.md#you-have-a1111-and-you-know-git). In this way, you can reuse all SD checkpoints and all extensions you installed previously in your OG SD-WebUI, but you should know what you are doing.36 37If you know what you are doing, you can install Forge using same method as SD-WebUI. (Install Git, Python, Git Clone the forge repo `https://github.com/lllyasviel/stable-diffusion-webui-forge.git` and then run webui-user.bat).38 39**Or you can just use this one-click installation package (with git and python included).**40 41[>>> Click Here to Download One-Click Package<<<](https://github.com/lllyasviel/stable-diffusion-webui-forge/releases/download/latest/webui_forge_cu121_torch21.7z)42 43After you download, you uncompress, use `update.bat` to update, and use `run.bat` to run.44 45Note that running `update.bat` is important, otherwise you may be using a previous version with potential bugs unfixed.46 4748 49# Screenshots of Comparison50 51I tested with several devices, and this is a typical result from 8GB VRAM (3070ti laptop) with SDXL.52 53**This is original WebUI:**54 5556 5758 5960 6162 63(average about 7.4GB/8GB, peak at about 7.9GB/8GB)64 65**This is WebUI Forge:**66 6768 6970 7172 7374 75(average and peak are all 6.3GB/8GB)76 77You can see that Forge does not change WebUI results. Installing Forge is not a seed breaking change. 78 79Forge can perfectly keep WebUI unchanged even for most complicated prompts like `fantasy landscape with a [mountain:lake:0.25] and [an oak:a christmas tree:0.75][ in foreground::0.6][ in background:0.25] [shoddy:masterful:0.5]`.80 81All your previous works still work in Forge!82 83# Forge Backend84 85Forge backend removes all WebUI's codes related to resource management and reworked everything. All previous CMD flags like `medvram, lowvram, medvram-sdxl, precision full, no half, no half vae, attention_xxx, upcast unet`, ... are all **REMOVED**. Adding these flags will not cause error but they will not do anything now. **We highly encourage Forge users to remove all cmd flags and let Forge to decide how to load models.**86 87Without any cmd flag, Forge can run SDXL with 4GB vram and SD1.5 with 2GB vram.88 89**Some flags that you may still pay attention to:** 90 911. `--always-offload-from-vram` (This flag will make things **slower** but less risky). This option will let Forge always unload models from VRAM. This can be useful if you use multiple software together and want Forge to use less VRAM and give some VRAM to other software, or when you are using some old extensions that will compete vram with Forge, or (very rarely) when you get OOM.92 932. `--cuda-malloc` (This flag will make things **faster** but more risky). This will ask pytorch to use *cudaMallocAsync* for tensor malloc. On some profilers I can observe performance gain at millisecond level, but the real speed up on most my devices are often unnoticed (about or less than 0.1 second per image). This cannot be set as default because many users reported issues that the async malloc will crash the program. Users need to enable this cmd flag at their own risk.94 953. `--cuda-stream` (This flag will make things **faster** but more risky). This will use pytorch CUDA streams (a special type of thread on GPU) to move models and compute tensors simultaneously. This can almost eliminate all model moving time, and speed up SDXL on 30XX/40XX devices with small VRAM (eg, RTX 4050 6GB, RTX 3060 Laptop 6GB, etc) by about 15\% to 25\%. However, this unfortunately cannot be set as default because I observe higher possibility of pure black images (Nan outputs) on 2060, and higher chance of OOM on 1080 and 2060. When the resolution is large, there is a chance that the computation time of one single attention layer is longer than the time for moving entire model to GPU. When that happens, the next attention layer will OOM since the GPU is filled with the entire model, and no remaining space is available for computing another attention layer. Most overhead detecting methods are not robust enough to be reliable on old devices (in my tests). Users need to enable this cmd flag at their own risk.96 974. `--pin-shared-memory` (This flag will make things **faster** but more risky). Effective only when used together with `--cuda-stream`. This will offload modules to Shared GPU Memory instead of system RAM when offloading models. On some 30XX/40XX devices with small VRAM (eg, RTX 4050 6GB, RTX 3060 Laptop 6GB, etc), I can observe significant (at least 20\%) speed-up for SDXL. However, this unfortunately cannot be set as default because the OOM of Shared GPU Memory is a much more severe problem than common GPU memory OOM. Pytorch does not provide any robust method to unload or detect Shared GPU Memory. Once the Shared GPU Memory OOM, the entire program will crash (observed with SDXL on GTX 1060/1050/1066), and there is no dynamic method to prevent or recover from the crash. Users need to enable this cmd flag at their own risk.98 99If you really want to play with cmd flags, you can additionally control the GPU with:100 101(extreme VRAM cases)102 103 --always-gpu104 --always-cpu105 106(rare attention cases)107 108 --attention-split109 --attention-quad110 --attention-pytorch111 --disable-xformers112 --disable-attention-upcast113 114(float point type)115 116 --all-in-fp32117 --all-in-fp16118 --unet-in-bf16119 --unet-in-fp16120 --unet-in-fp8-e4m3fn121 --unet-in-fp8-e5m2122 --vae-in-fp16123 --vae-in-fp32124 --vae-in-bf16125 --clip-in-fp8-e4m3fn126 --clip-in-fp8-e5m2127 --clip-in-fp16128 --clip-in-fp32129 130(rare platforms)131 132 --directml133 --disable-ipex-hijack134 --pytorch-deterministic135 136Again, Forge do not recommend users to use any cmd flags unless you are very sure that you really need these.137 138# UNet Patcher139 140Note that [Forge does not use any other software as backend](https://github.com/lllyasviel/stable-diffusion-webui-forge/discussions/169). The full name of the backend is `Stable Diffusion WebUI with Forge backend`, or for simplicity, the `Forge backend`. The API and python symbols are made similar to previous software only for reducing the learning cost of developers.141 142Now developing an extension is super simple. We finally have a patchable UNet.143 144Below is using one single file with 80 lines of codes to support FreeU:145 146`extensions-builtin/sd_forge_freeu/scripts/forge_freeu.py`147 148```python149import torch150import gradio as gr151from modules import scripts152 153 154def Fourier_filter(x, threshold, scale):155 x_freq = torch.fft.fftn(x.float(), dim=(-2, -1))156 x_freq = torch.fft.fftshift(x_freq, dim=(-2, -1))157 B, C, H, W = x_freq.shape158 mask = torch.ones((B, C, H, W), device=x.device)159 crow, ccol = H // 2, W //2160 mask[..., crow - threshold:crow + threshold, ccol - threshold:ccol + threshold] = scale161 x_freq = x_freq * mask162 x_freq = torch.fft.ifftshift(x_freq, dim=(-2, -1))163 x_filtered = torch.fft.ifftn(x_freq, dim=(-2, -1)).real164 return x_filtered.to(x.dtype)165 166 167def set_freeu_v2_patch(model, b1, b2, s1, s2):168 model_channels = model.model.model_config.unet_config["model_channels"]169 scale_dict = {model_channels * 4: (b1, s1), model_channels * 2: (b2, s2)}170 171 def output_block_patch(h, hsp, *args, **kwargs):172 scale = scale_dict.get(h.shape[1], None)173 if scale is not None:174 hidden_mean = h.mean(1).unsqueeze(1)175 B = hidden_mean.shape[0]176 hidden_max, _ = torch.max(hidden_mean.view(B, -1), dim=-1, keepdim=True)177 hidden_min, _ = torch.min(hidden_mean.view(B, -1), dim=-1, keepdim=True)178 hidden_mean = (hidden_mean - hidden_min.unsqueeze(2).unsqueeze(3)) / \179 (hidden_max - hidden_min).unsqueeze(2).unsqueeze(3)180 h[:, :h.shape[1] // 2] = h[:, :h.shape[1] // 2] * ((scale[0] - 1) * hidden_mean + 1)181 hsp = Fourier_filter(hsp, threshold=1, scale=scale[1])182 return h, hsp183 184 m = model.clone()185 m.set_model_output_block_patch(output_block_patch)186 return m187 188 189class FreeUForForge(scripts.Script):190 def title(self):191 return "FreeU Integrated"192 193 def show(self, is_img2img):194 # make this extension visible in both txt2img and img2img tab.195 return scripts.AlwaysVisible196 197 def ui(self, *args, **kwargs):198 with gr.Accordion(open=False, label=self.title()):199 freeu_enabled = gr.Checkbox(label='Enabled', value=False)200 freeu_b1 = gr.Slider(label='B1', minimum=0, maximum=2, step=0.01, value=1.01)201 freeu_b2 = gr.Slider(label='B2', minimum=0, maximum=2, step=0.01, value=1.02)202 freeu_s1 = gr.Slider(label='S1', minimum=0, maximum=4, step=0.01, value=0.99)203 freeu_s2 = gr.Slider(label='S2', minimum=0, maximum=4, step=0.01, value=0.95)204 205 return freeu_enabled, freeu_b1, freeu_b2, freeu_s1, freeu_s2206 207 def process_before_every_sampling(self, p, *script_args, **kwargs):208 # This will be called before every sampling.209 # If you use highres fix, this will be called twice.210 211 freeu_enabled, freeu_b1, freeu_b2, freeu_s1, freeu_s2 = script_args212 213 if not freeu_enabled:214 return215 216 unet = p.sd_model.forge_objects.unet217 218 unet = set_freeu_v2_patch(unet, freeu_b1, freeu_b2, freeu_s1, freeu_s2)219 220 p.sd_model.forge_objects.unet = unet221 222 # Below codes will add some logs to the texts below the image outputs on UI.223 # The extra_generation_params does not influence results.224 p.extra_generation_params.update(dict(225 freeu_enabled=freeu_enabled,226 freeu_b1=freeu_b1,227 freeu_b2=freeu_b2,228 freeu_s1=freeu_s1,229 freeu_s2=freeu_s2,230 ))231 232 return233```234 235It looks like this:236 237238 239Similar components like HyperTile, KohyaHighResFix, SAG, can all be implemented within 100 lines of codes (see also the codes).240 241242 243ControlNets can finally be called by different extensions.244 245Implementing Stable Video Diffusion and Zero123 are also super simple now (see also the codes). 246 247*Stable Video Diffusion:*248 249`extensions-builtin/sd_forge_svd/scripts/forge_svd.py`250 251```python252import torch253import gradio as gr254import os255import pathlib256 257from modules import script_callbacks258from modules.paths import models_path259from modules.ui_common import ToolButton, refresh_symbol260from modules import shared261 262from modules_forge.forge_util import numpy_to_pytorch, pytorch_to_numpy263from ldm_patched.modules.sd import load_checkpoint_guess_config264from ldm_patched.contrib.external_video_model import VideoLinearCFGGuidance, SVD_img2vid_Conditioning265from ldm_patched.contrib.external import KSampler, VAEDecode266 267 268opVideoLinearCFGGuidance = VideoLinearCFGGuidance()269opSVD_img2vid_Conditioning = SVD_img2vid_Conditioning()270opKSampler = KSampler()271opVAEDecode = VAEDecode()272 273svd_root = os.path.join(models_path, 'svd')274os.makedirs(svd_root, exist_ok=True)275svd_filenames = []276 277 278def update_svd_filenames():279 global svd_filenames280 svd_filenames = [281 pathlib.Path(x).name for x in282 shared.walk_files(svd_root, allowed_extensions=[".pt", ".ckpt", ".safetensors"])283 ]284 return svd_filenames285 286 287@torch.inference_mode()288@torch.no_grad()289def predict(filename, width, height, video_frames, motion_bucket_id, fps, augmentation_level,290 sampling_seed, sampling_steps, sampling_cfg, sampling_sampler_name, sampling_scheduler,291 sampling_denoise, guidance_min_cfg, input_image):292 filename = os.path.join(svd_root, filename)293 model_raw, _, vae, clip_vision = \294 load_checkpoint_guess_config(filename, output_vae=True, output_clip=False, output_clipvision=True)295 model = opVideoLinearCFGGuidance.patch(model_raw, guidance_min_cfg)[0]296 init_image = numpy_to_pytorch(input_image)297 positive, negative, latent_image = opSVD_img2vid_Conditioning.encode(298 clip_vision, init_image, vae, width, height, video_frames, motion_bucket_id, fps, augmentation_level)299 output_latent = opKSampler.sample(model, sampling_seed, sampling_steps, sampling_cfg,300 sampling_sampler_name, sampling_scheduler, positive,301 negative, latent_image, sampling_denoise)[0]302 output_pixels = opVAEDecode.decode(vae, output_latent)[0]303 outputs = pytorch_to_numpy(output_pixels)304 return outputs305 306 307def on_ui_tabs():308 with gr.Blocks() as svd_block:309 with gr.Row():310 with gr.Column():311 input_image = gr.Image(label='Input Image', source='upload', type='numpy', height=400)312 313 with gr.Row():314 filename = gr.Dropdown(label="SVD Checkpoint Filename",315 choices=svd_filenames,316 value=svd_filenames[0] if len(svd_filenames) > 0 else None)317 refresh_button = ToolButton(value=refresh_symbol, tooltip="Refresh")318 refresh_button.click(319 fn=lambda: gr.update(choices=update_svd_filenames),320 inputs=[], outputs=filename)321 322 width = gr.Slider(label='Width', minimum=16, maximum=8192, step=8, value=1024)323 height = gr.Slider(label='Height', minimum=16, maximum=8192, step=8, value=576)324 video_frames = gr.Slider(label='Video Frames', minimum=1, maximum=4096, step=1, value=14)325 motion_bucket_id = gr.Slider(label='Motion Bucket Id', minimum=1, maximum=1023, step=1, value=127)326 fps = gr.Slider(label='Fps', minimum=1, maximum=1024, step=1, value=6)327 augmentation_level = gr.Slider(label='Augmentation Level', minimum=0.0, maximum=10.0, step=0.01,328 value=0.0)329 sampling_steps = gr.Slider(label='Sampling Steps', minimum=1, maximum=200, step=1, value=20)330 sampling_cfg = gr.Slider(label='CFG Scale', minimum=0.0, maximum=50.0, step=0.1, value=2.5)331 sampling_denoise = gr.Slider(label='Sampling Denoise', minimum=0.0, maximum=1.0, step=0.01, value=1.0)332 guidance_min_cfg = gr.Slider(label='Guidance Min Cfg', minimum=0.0, maximum=100.0, step=0.5, value=1.0)333 sampling_sampler_name = gr.Radio(label='Sampler Name',334 choices=['euler', 'euler_ancestral', 'heun', 'heunpp2', 'dpm_2',335 'dpm_2_ancestral', 'lms', 'dpm_fast', 'dpm_adaptive',336 'dpmpp_2s_ancestral', 'dpmpp_sde', 'dpmpp_sde_gpu',337 'dpmpp_2m', 'dpmpp_2m_sde', 'dpmpp_2m_sde_gpu',338 'dpmpp_3m_sde', 'dpmpp_3m_sde_gpu', 'ddpm', 'lcm', 'ddim',339 'uni_pc', 'uni_pc_bh2'], value='euler')340 sampling_scheduler = gr.Radio(label='Scheduler',341 choices=['normal', 'karras', 'exponential', 'sgm_uniform', 'simple',342 'ddim_uniform'], value='karras')343 sampling_seed = gr.Number(label='Seed', value=12345, precision=0)344 345 generate_button = gr.Button(value="Generate")346 347 ctrls = [filename, width, height, video_frames, motion_bucket_id, fps, augmentation_level,348 sampling_seed, sampling_steps, sampling_cfg, sampling_sampler_name, sampling_scheduler,349 sampling_denoise, guidance_min_cfg, input_image]350 351 with gr.Column():352 output_gallery = gr.Gallery(label='Gallery', show_label=False, object_fit='contain',353 visible=True, height=1024, columns=4)354 355 generate_button.click(predict, inputs=ctrls, outputs=[output_gallery])356 return [(svd_block, "SVD", "svd")]357 358 359update_svd_filenames()360script_callbacks.on_ui_tabs(on_ui_tabs)361```362 363Note that although the above codes look like independent codes, they actually will automatically offload/unload any other models. For example, below is me opening webui, load SDXL, generated an image, then go to SVD, then generated image frames. You can see that the GPU memory is perfectly managed and the SDXL is moved to RAM then SVD is moved to GPU. 364 365Note that this management is fully automatic. This makes writing extensions super simple.366 367368 369370 371Similarly, Zero123:372 373374 375### Write a simple ControlNet:376 377Below is a simple extension to have a completely independent pass of ControlNet that never conflicts any other extensions:378 379`extensions-builtin/sd_forge_controlnet_example/scripts/sd_forge_controlnet_example.py`380 381Note that this extension is hidden because it is only for developers. To see it in UI, use `--show-controlnet-example`.382 383The memory optimization in this example is fully automatic. You do not need to care about memory and inference speed, but you may want to cache objects if you wish.384 385```python386# Use --show-controlnet-example to see this extension.387 388import cv2389import gradio as gr390import torch391 392from modules import scripts393from modules.shared_cmd_options import cmd_opts394from modules_forge.shared import supported_preprocessors395from modules.modelloader import load_file_from_url396from ldm_patched.modules.controlnet import load_controlnet397from modules_forge.controlnet import apply_controlnet_advanced398from modules_forge.forge_util import numpy_to_pytorch399from modules_forge.shared import controlnet_dir400 401 402class ControlNetExampleForge(scripts.Script):403 model = None404 405 def title(self):406 return "ControlNet Example for Developers"407 408 def show(self, is_img2img):409 # make this extension visible in both txt2img and img2img tab.410 return scripts.AlwaysVisible411 412 def ui(self, *args, **kwargs):413 with gr.Accordion(open=False, label=self.title()):414 gr.HTML('This is an example controlnet extension for developers.')415 gr.HTML('You see this extension because you used --show-controlnet-example')416 input_image = gr.Image(source='upload', type='numpy')417 funny_slider = gr.Slider(label='This slider does nothing. It just shows you how to transfer parameters.',418 minimum=0.0, maximum=1.0, value=0.5)419 420 return input_image, funny_slider421 422 def process(self, p, *script_args, **kwargs):423 input_image, funny_slider = script_args424 425 # This slider does nothing. It just shows you how to transfer parameters.426 del funny_slider427 428 if input_image is None:429 return430 431 # controlnet_canny_path = load_file_from_url(432 # url='https://huggingface.co/lllyasviel/sd_control_collection/resolve/main/sai_xl_canny_256lora.safetensors',433 # model_dir=model_dir,434 # file_name='sai_xl_canny_256lora.safetensors'435 # )436 controlnet_canny_path = load_file_from_url(437 url='https://huggingface.co/lllyasviel/fav_models/resolve/main/fav/control_v11p_sd15_canny_fp16.safetensors',438 model_dir=controlnet_dir,439 file_name='control_v11p_sd15_canny_fp16.safetensors'440 )441 print('The model [control_v11p_sd15_canny_fp16.safetensors] download finished.')442 443 self.model = load_controlnet(controlnet_canny_path)444 print('Controlnet loaded.')445 446 return447 448 def process_before_every_sampling(self, p, *script_args, **kwargs):449 # This will be called before every sampling.450 # If you use highres fix, this will be called twice.451 452 input_image, funny_slider = script_args453 454 if input_image is None or self.model is None:455 return456 457 B, C, H, W = kwargs['noise'].shape # latent_shape458 height = H * 8459 width = W * 8460 batch_size = p.batch_size461 462 preprocessor = supported_preprocessors['canny']463 464 # detect control at certain resolution465 control_image = preprocessor(466 input_image, resolution=512, slider_1=100, slider_2=200, slider_3=None)467 468 # here we just use nearest neighbour to align input shape.469 # You may want crop and resize, or crop and fill, or others.470 control_image = cv2.resize(471 control_image, (width, height), interpolation=cv2.INTER_NEAREST)472 473 # Output preprocessor result. Now called every sampling. Cache in your own way.474 p.extra_result_images.append(control_image)475 476 print('Preprocessor Canny finished.')477 478 control_image_bchw = numpy_to_pytorch(control_image).movedim(-1, 1)479 480 unet = p.sd_model.forge_objects.unet481 482 # Unet has input, middle, output blocks, and we can give different weights483 # to each layers in all blocks.484 # Below is an example for stronger control in middle block.485 # This is helpful for some high-res fix passes. (p.is_hr_pass)486 positive_advanced_weighting = {487 'input': [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.1, 1.2],488 'middle': [1.0],489 'output': [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.1, 1.2]490 }491 negative_advanced_weighting = {492 'input': [0.15, 0.25, 0.35, 0.45, 0.55, 0.65, 0.75, 0.85, 0.95, 1.05, 1.15, 1.25],493 'middle': [1.05],494 'output': [0.15, 0.25, 0.35, 0.45, 0.55, 0.65, 0.75, 0.85, 0.95, 1.05, 1.15, 1.25]495 }496 497 # The advanced_frame_weighting is a weight applied to each image in a batch.498 # The length of this list must be same with batch size499 # For example, if batch size is 5, the below list is [0.2, 0.4, 0.6, 0.8, 1.0]500 # If you view the 5 images as 5 frames in a video, this will lead to501 # progressively stronger control over time.502 advanced_frame_weighting = [float(i + 1) / float(batch_size) for i in range(batch_size)]503 504 # The advanced_sigma_weighting allows you to dynamically compute control505 # weights given diffusion timestep (sigma).506 # For example below code can softly make beginning steps stronger than ending steps.507 sigma_max = unet.model.model_sampling.sigma_max508 sigma_min = unet.model.model_sampling.sigma_min509 advanced_sigma_weighting = lambda s: (s - sigma_min) / (sigma_max - sigma_min)510 511 # You can even input a tensor to mask all control injections512 # The mask will be automatically resized during inference in UNet.513 # The size should be B 1 H W and the H and W are not important514 # because they will be resized automatically515 advanced_mask_weighting = torch.ones(size=(1, 1, 512, 512))516 517 # But in this simple example we do not use them518 positive_advanced_weighting = None519 negative_advanced_weighting = None520 advanced_frame_weighting = None521 advanced_sigma_weighting = None522 advanced_mask_weighting = None523 524 unet = apply_controlnet_advanced(unet=unet, controlnet=self.model, image_bchw=control_image_bchw,525 strength=0.6, start_percent=0.0, end_percent=0.8,526 positive_advanced_weighting=positive_advanced_weighting,527 negative_advanced_weighting=negative_advanced_weighting,528 advanced_frame_weighting=advanced_frame_weighting,529 advanced_sigma_weighting=advanced_sigma_weighting,530 advanced_mask_weighting=advanced_mask_weighting)531 532 p.sd_model.forge_objects.unet = unet533 534 # Below codes will add some logs to the texts below the image outputs on UI.535 # The extra_generation_params does not influence results.536 p.extra_generation_params.update(dict(537 controlnet_info='You should see these texts below output images!',538 ))539 540 return541 542 543# Use --show-controlnet-example to see this extension.544if not cmd_opts.show_controlnet_example:545 del ControlNetExampleForge546 547```548 549550 551 552### Add a preprocessor553 554Below is the full codes to add a normalbae preprocessor with perfect memory managements.555 556You can use arbitrary independent extensions to add a preprocessor.557 558Your preprocessor will be read by all other extensions using `modules_forge.shared.preprocessors`559 560Below codes are in `extensions-builtin\forge_preprocessor_normalbae\scripts\preprocessor_normalbae.py`561 562```python563from modules_forge.supported_preprocessor import Preprocessor, PreprocessorParameter564from modules_forge.shared import preprocessor_dir, add_supported_preprocessor565from modules_forge.forge_util import resize_image_with_pad566from modules.modelloader import load_file_from_url567 568import types569import torch570import numpy as np571 572from einops import rearrange573from annotator.normalbae.models.NNET import NNET574from annotator.normalbae import load_checkpoint575from torchvision import transforms576 577 578class PreprocessorNormalBae(Preprocessor):579 def __init__(self):580 super().__init__()581 self.name = 'normalbae'582 self.tags = ['NormalMap']583 self.model_filename_filters = ['normal']584 self.slider_resolution = PreprocessorParameter(585 label='Resolution', minimum=128, maximum=2048, value=512, step=8, visible=True)586 self.slider_1 = PreprocessorParameter(visible=False)587 self.slider_2 = PreprocessorParameter(visible=False)588 self.slider_3 = PreprocessorParameter(visible=False)589 self.show_control_mode = True590 self.do_not_need_model = False591 self.sorting_priority = 100 # higher goes to top in the list592 593 def load_model(self):594 if self.model_patcher is not None:595 return596 597 model_path = load_file_from_url(598 "https://huggingface.co/lllyasviel/Annotators/resolve/main/scannet.pt",599 model_dir=preprocessor_dir)600 601 args = types.SimpleNamespace()602 args.mode = 'client'603 args.architecture = 'BN'604 args.pretrained = 'scannet'605 args.sampling_ratio = 0.4606 args.importance_ratio = 0.7607 model = NNET(args)608 model = load_checkpoint(model_path, model)609 self.norm = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])610 611 self.model_patcher = self.setup_model_patcher(model)612 613 def __call__(self, input_image, resolution, slider_1=None, slider_2=None, slider_3=None, **kwargs):614 input_image, remove_pad = resize_image_with_pad(input_image, resolution)615 616 self.load_model()617 618 self.move_all_model_patchers_to_gpu()619 620 assert input_image.ndim == 3621 image_normal = input_image622 623 with torch.no_grad():624 image_normal = self.send_tensor_to_model_device(torch.from_numpy(image_normal))625 image_normal = image_normal / 255.0626 image_normal = rearrange(image_normal, 'h w c -> 1 c h w')627 image_normal = self.norm(image_normal)628 629 normal = self.model_patcher.model(image_normal)630 normal = normal[0][-1][:, :3]631 normal = ((normal + 1) * 0.5).clip(0, 1)632 633 normal = rearrange(normal[0], 'c h w -> h w c').cpu().numpy()634 normal_image = (normal * 255.0).clip(0, 255).astype(np.uint8)635 636 return remove_pad(normal_image)637 638 639add_supported_preprocessor(PreprocessorNormalBae())640 641```642 643# New features (that are not available in original WebUI)644 645Thanks to Unet Patcher, many new things are possible now and supported in Forge, including SVD, Z123, masked Ip-adapter, masked controlnet, photomaker, etc.646 647Masked Ip-Adapter648 649650 651652 653654 655Masked ControlNet656 657658 659660 661662 663PhotoMaker664 665(Note that photomaker is a special control that need you to add the trigger word "photomaker". Your prompt should be like "a photo of photomaker")666 667668 669Marigold Depth670 671672 673# New Samplers (that are not in origin)674 675 DDPM676 DDPM Karras677 DPM++ 2M Turbo678 DPM++ 2M SDE Turbo679 LCM Karras680 Euler A Turbo681 682# About Extensions683 684ControlNet and TiledVAE are integrated, and you should uninstall these two extensions:685 686 sd-webui-controlnet687 multidiffusion-upscaler-for-automatic1111688 689Note that **AnimateDiff** is under construction by [continue-revolution](https://github.com/continue-revolution) at [sd-webui-animatediff forge/master branch](https://github.com/continue-revolution/sd-webui-animatediff/tree/forge/master) and [sd-forge-animatediff](https://github.com/continue-revolution/sd-forge-animatediff) (they are in sync). (continue-revolution original words: prompt travel, inf t2v, controlnet v2v have been proven to work well; motion lora, i2i batch still under construction and may be finished in a week")690 691Other extensions should work without problems, like:692 693 canvas-zoom694 translations/localizations695 Dynamic Prompts696 Adetailer697 Ultimate SD Upscale698 Reactor699 700However, if newer extensions use Forge, their codes can be much shorter. 701 702Usually if an old extension rework using Forge's unet patcher, 80% codes can be removed, especially when they need to call controlnet.703 704# Contribution705 706Forge uses a bot to get commits and codes from https://github.com/AUTOMATIC1111/stable-diffusion-webui/tree/dev every afternoon (if merge is automatically successful by a git bot, or by my compiler, or by my ChatGPT bot) or mid-night (if my compiler and my ChatGPT bot both failed to merge and I review it manually).707 708All PRs that can be implemented in https://github.com/AUTOMATIC1111/stable-diffusion-webui/tree/dev should submit PRs there.709 710Feel free to submit PRs related to the functionality of Forge here.711