diffusers/controlnet-depth-sdxl-1.0-mid

1 2---3license: openrail++4base_model: stabilityai/stable-diffusion-xl-base-1.05tags:6- stable-diffusion-xl7- stable-diffusion-xl-diffusers8- text-to-image9- diffusers10- controlnet11inference: false12---13    14# SDXL-controlnet: Depth15 16These are controlnet weights trained on stabilityai/stable-diffusion-xl-base-1.0 with depth conditioning. This checkpoint is 5x smaller than the original XL controlnet checkpoint. You can find some example images in the following.17 18prompt: donald trump, serious look, cigar in the mouth, 70mm, film still, head shot19![open](oppenheimer_mid.png)20 21prompt: spiderman lecture, photorealistic22![images_0)](./spiderman_mid.png)23 24prompt: aerial view, a futuristic research complex in a bright foggy jungle, hard lighting25![images_1)](./hf_logo_mid.png)26 27prompt: megatron in an apocalyptic world ground, runied city in the background, photorealistic28![images_2)](./megatron_mid.png)29 30## Usage31 32Make sure to first install the libraries:33 34```bash35pip install accelerate transformers safetensors diffusers36```37 38And then we're ready to go:39 40```python41import torch42import numpy as np43from PIL import Image44 45from transformers import DPTFeatureExtractor, DPTForDepthEstimation46from diffusers import ControlNetModel, StableDiffusionXLControlNetPipeline, AutoencoderKL47from diffusers.utils import load_image48 49 50depth_estimator = DPTForDepthEstimation.from_pretrained("Intel/dpt-hybrid-midas").to("cuda")51feature_extractor = DPTFeatureExtractor.from_pretrained("Intel/dpt-hybrid-midas")52controlnet = ControlNetModel.from_pretrained(53    "diffusers/controlnet-depth-sdxl-1.0-mid",54    variant="fp16",55    use_safetensors=True,56    torch_dtype=torch.float16,57).to("cuda")58vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16).to("cuda")59pipe = StableDiffusionXLControlNetPipeline.from_pretrained(60    "stabilityai/stable-diffusion-xl-base-1.0",61    controlnet=controlnet,62    vae=vae,63    variant="fp16",64    use_safetensors=True,65    torch_dtype=torch.float16,66).to("cuda")67pipe.enable_model_cpu_offload()68 69def get_depth_map(image):70    image = feature_extractor(images=image, return_tensors="pt").pixel_values.to("cuda")71    with torch.no_grad(), torch.autocast("cuda"):72        depth_map = depth_estimator(image).predicted_depth73 74    depth_map = torch.nn.functional.interpolate(75        depth_map.unsqueeze(1),76        size=(1024, 1024),77        mode="bicubic",78        align_corners=False,79    )80    depth_min = torch.amin(depth_map, dim=[1, 2, 3], keepdim=True)81    depth_max = torch.amax(depth_map, dim=[1, 2, 3], keepdim=True)82    depth_map = (depth_map - depth_min) / (depth_max - depth_min)83    image = torch.cat([depth_map] * 3, dim=1)84 85    image = image.permute(0, 2, 3, 1).cpu().numpy()[0]86    image = Image.fromarray((image * 255.0).clip(0, 255).astype(np.uint8))87    return image88 89 90prompt = "stormtrooper lecture, photorealistic"91image = load_image("https://huggingface.co/lllyasviel/sd-controlnet-depth/resolve/main/images/stormtrooper.png")92controlnet_conditioning_scale = 0.5  # recommended for good generalization93 94depth_image = get_depth_map(image)95 96images = pipe(97    prompt, image=depth_image, num_inference_steps=30, controlnet_conditioning_scale=controlnet_conditioning_scale,98).images99images[0]100 101images[0].save(f"stormtrooper_grid.png")102```103 104![](stormtrooper_grid.png)105 106To more details, check out the official documentation of [`StableDiffusionXLControlNetPipeline`](https://huggingface.co/docs/diffusers/main/en/api/pipelines/controlnet_sdxl).107 108🚨 Please note that this checkpoint is experimental and there's a lot of room for improvement. We encourage the community to build on top of it, improve it, and provide us with feedback. 🚨109 110### Training111 112Our training script was built on top of the official training script that we provide [here](https://github.com/huggingface/diffusers/blob/main/examples/controlnet/README_sdxl.md). 113You can refer to [this script](https://github.com/huggingface/diffusers/blob/7b93c2a882d8e12209fbaeffa51ee2b599ab5349/examples/research_projects/controlnet/train_controlnet_webdataset.py) for full discolsure.114 115* This checkpoint does not perform distillation. We just use a smaller ControlNet initialized from the SDXL UNet. We116encourage the community to try and conduct distillation too. This resource might be of help in [this regard](https://huggingface.co/blog/sd_distillation). 117* To learn more about how the ControlNet was initialized, refer to [this code block](https://github.com/huggingface/diffusers/blob/7b93c2a882d8e12209fbaeffa51ee2b599ab5349/examples/research_projects/controlnet/train_controlnet_webdataset.py#L981C1-L999C36). 118* It does not have any attention blocks.119* The model works pretty good on most conditioning images. But for more complex conditionings, the bigger checkpoints might be better. We are still working on improving the quality of this checkpoint and looking for feedback from the community.120* We recommend playing around with the `controlnet_conditioning_scale` and `guidance_scale` arguments for potentially better121image generation quality.122 123#### Training data124The model was trained on 3M images from LAION aesthetic 6 plus subset, with batch size of 256 for 50k steps with constant learning rate of 3e-5.125 126#### Compute127One 8xA100 machine128 129#### Mixed precision130FP16