ByteDance/SDXL-Lightning
2.2k87k
1---2license: openrail++3tags:4- text-to-image5- stable-diffusion6library_name: diffusers7inference: false8---9 10# SDXL-Lightning11 1213 14SDXL-Lightning is a lightning-fast text-to-image generation model. It can generate high-quality 1024px images in a few steps. For more information, please refer to our research paper: [SDXL-Lightning: Progressive Adversarial Diffusion Distillation](https://arxiv.org/abs/2402.13929). We open-source the model as part of the research.15 16Our models are distilled from [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0). This repository contains checkpoints for 1-step, 2-step, 4-step, and 8-step distilled models. The generation quality of our 2-step, 4-step, and 8-step model is amazing. Our 1-step model is more experimental.17 18We provide both full UNet and LoRA checkpoints. The full UNet models have the best quality while the LoRA models can be applied to other base models.19 20## Demos21 22* Generate with all configurations, best quality: [Demo](https://huggingface.co/spaces/ByteDance/SDXL-Lightning)23 24## Checkpoints25 26* `sdxl_lightning_Nstep.safetensors`: All-in-one checkpoint, for ComfyUI.27* `sdxl_lightning_Nstep_unet.safetensors`: UNet checkpoint only, for Diffusers.28* `sdxl_lightning_Nstep_lora.safetensors`: LoRA checkpoint, for Diffusers and ComfyUI.29 30## Diffusers Usage31 32Please always use the correct checkpoint for the corresponding inference steps.33 34### 2-Step, 4-Step, 8-Step UNet35 36```python37import torch38from diffusers import StableDiffusionXLPipeline, UNet2DConditionModel, EulerDiscreteScheduler39from huggingface_hub import hf_hub_download40from safetensors.torch import load_file41 42base = "stabilityai/stable-diffusion-xl-base-1.0"43repo = "ByteDance/SDXL-Lightning"44ckpt = "sdxl_lightning_4step_unet.safetensors" # Use the correct ckpt for your step setting!45 46# Load model.47unet = UNet2DConditionModel.from_config(base, subfolder="unet").to("cuda", torch.float16)48unet.load_state_dict(load_file(hf_hub_download(repo, ckpt), device="cuda"))49pipe = StableDiffusionXLPipeline.from_pretrained(base, unet=unet, torch_dtype=torch.float16, variant="fp16").to("cuda")50 51# Ensure sampler uses "trailing" timesteps.52pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config, timestep_spacing="trailing")53 54# Ensure using the same inference steps as the loaded model and CFG set to 0.55pipe("A girl smiling", num_inference_steps=4, guidance_scale=0).images[0].save("output.png")56```57 58### 2-Step, 4-Step, 8-Step LoRA59 60Use LoRA only if you are using non-SDXL base models. Otherwise use our UNet checkpoint for better quality.61```python62import torch63from diffusers import StableDiffusionXLPipeline, EulerDiscreteScheduler64from huggingface_hub import hf_hub_download65 66base = "stabilityai/stable-diffusion-xl-base-1.0"67repo = "ByteDance/SDXL-Lightning"68ckpt = "sdxl_lightning_4step_lora.safetensors" # Use the correct ckpt for your step setting!69 70# Load model.71pipe = StableDiffusionXLPipeline.from_pretrained(base, torch_dtype=torch.float16, variant="fp16").to("cuda")72pipe.load_lora_weights(hf_hub_download(repo, ckpt))73pipe.fuse_lora()74 75# Ensure sampler uses "trailing" timesteps.76pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config, timestep_spacing="trailing")77 78# Ensure using the same inference steps as the loaded model and CFG set to 0.79pipe("A girl smiling", num_inference_steps=4, guidance_scale=0).images[0].save("output.png")80```81 82### 1-Step UNet83The 1-step model is only experimental and the quality is much less stable. Consider using the 2-step model for much better quality.84 85The 1-step model uses "sample" prediction instead of "epsilon" prediction! The scheduler needs to be configured correctly.86 87```python88import torch89from diffusers import StableDiffusionXLPipeline, UNet2DConditionModel, EulerDiscreteScheduler90from huggingface_hub import hf_hub_download91from safetensors.torch import load_file92 93base = "stabilityai/stable-diffusion-xl-base-1.0"94repo = "ByteDance/SDXL-Lightning"95ckpt = "sdxl_lightning_1step_unet_x0.safetensors" # Use the correct ckpt for your step setting!96 97# Load model.98unet = UNet2DConditionModel.from_config(base, subfolder="unet").to("cuda", torch.float16)99unet.load_state_dict(load_file(hf_hub_download(repo, ckpt), device="cuda"))100pipe = StableDiffusionXLPipeline.from_pretrained(base, unet=unet, torch_dtype=torch.float16, variant="fp16").to("cuda")101 102# Ensure sampler uses "trailing" timesteps and "sample" prediction type.103pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config, timestep_spacing="trailing", prediction_type="sample")104 105# Ensure using the same inference steps as the loaded model and CFG set to 0.106pipe("A girl smiling", num_inference_steps=1, guidance_scale=0).images[0].save("output.png")107```108 109 110## ComfyUI Usage111 112Please always use the correct checkpoint for the corresponding inference steps.113Please use Euler sampler with sgm_uniform scheduler.114 115### 2-Step, 4-Step, 8-Step Full116 1171. Download the full checkpoint (`sdxl_lightning_Nstep.safetensors`) to `/ComfyUI/models/checkpoints`.1181. Download our [ComfyUI full workflow](comfyui/sdxl_lightning_workflow_full.json).119 120121 122### 2-Step, 4-Step, 8-Step LoRA123 124Use LoRA only if you are using non-SDXL base models. Otherwise use our full checkpoint for better quality.125 1261. Prepare your own base model.1271. Download the LoRA checkpoint (`sdxl_lightning_Nstep_lora.safetensors`) to `/ComfyUI/models/loras`1281. Download our [ComfyUI LoRA workflow](comfyui/sdxl_lightning_workflow_lora.json).129 130131 132### 1-Step133 134The 1-step model is only experimental and the quality is much less stable. Consider using the 2-step model for much better quality.135 1361. Update your ComfyUI to the latest version.1371. Download the full checkpoint (`sdxl_lightning_1step_x0.safetensors`) to `/ComfyUI/models/checkpoints`.1381. Download our [ComfyUI full 1-step workflow](comfyui/sdxl_lightning_workflow_full_1step.json).139 140141 142 143## Cite Our Work144```145@misc{lin2024sdxllightning,146 title={SDXL-Lightning: Progressive Adversarial Diffusion Distillation}, 147 author={Shanchuan Lin and Anran Wang and Xiao Yang},148 year={2024},149 eprint={2402.13929},150 archivePrefix={arXiv},151 primaryClass={cs.CV}152}153```