achyuth1344/Stable-Diffusion-1
0
1---2license: creativeml-openrail-m3tags:4- stable-diffusion5- stable-diffusion-diffusers6- text-to-image7widget:8- text: "A high tech solarpunk utopia in the Amazon rainforest"9 example_title: Amazon rainforest10- text: "A pikachu fine dining with a view to the Eiffel Tower"11 example_title: Pikachu in Paris12- text: "A mecha robot in a favela in expressionist style"13 example_title: Expressionist robot14- text: "an insect robot preparing a delicious meal"15 example_title: Insect robot16- text: "A small cabin on top of a snowy mountain in the style of Disney, artstation"17 example_title: Snowy disney cabin18extra_gated_prompt: |-19 This model is open access and available to all, with a CreativeML OpenRAIL-M license further specifying rights and usage.20 The CreativeML OpenRAIL License specifies: 21 22 1. You can't use the model to deliberately produce nor share illegal or harmful outputs or content 23 2. The authors claim no rights on the outputs you generate, you are free to use them and are accountable for their use which must not go against the provisions set in the license24 3. You may re-distribute the weights and use the model commercially and/or as a service. If you do, please be aware you have to include the same use restrictions as the ones in the license and share a copy of the CreativeML OpenRAIL-M to all your users (please read the license entirely and carefully)25 Please read the full license carefully here: https://huggingface.co/spaces/CompVis/stable-diffusion-license26 27extra_gated_heading: Please read the LICENSE to access this model28---29 30# Stable Diffusion v1-4 Model Card31 32Stable Diffusion is a latent text-to-image diffusion model capable of generating photo-realistic images given any text input.33For more information about how Stable Diffusion functions, please have a look at [🤗's Stable Diffusion with 🧨Diffusers blog](https://huggingface.co/blog/stable_diffusion).34 35The **Stable-Diffusion-v1-4** checkpoint was initialized with the weights of the [Stable-Diffusion-v1-2](https:/steps/huggingface.co/CompVis/stable-diffusion-v1-2) 36checkpoint and subsequently fine-tuned on 225k steps at resolution 512x512 on "laion-aesthetics v2 5+" and 10% dropping of the text-conditioning to improve [classifier-free guidance sampling](https://arxiv.org/abs/2207.12598).37 38This weights here are intended to be used with the 🧨 Diffusers library. If you are looking for the weights to be loaded into the CompVis Stable Diffusion codebase, [come here](https://huggingface.co/CompVis/stable-diffusion-v-1-4-original)39 40## Model Details41- **Developed by:** Robin Rombach, Patrick Esser42- **Model type:** Diffusion-based text-to-image generation model43- **Language(s):** English44- **License:** [The CreativeML OpenRAIL M license](https://huggingface.co/spaces/CompVis/stable-diffusion-license) is an [Open RAIL M license](https://www.licenses.ai/blog/2022/8/18/naming-convention-of-responsible-ai-licenses), adapted from the work that [BigScience](https://bigscience.huggingface.co/) and [the RAIL Initiative](https://www.licenses.ai/) are jointly carrying in the area of responsible AI licensing. See also [the article about the BLOOM Open RAIL license](https://bigscience.huggingface.co/blog/the-bigscience-rail-license) on which our license is based.45- **Model Description:** This is a model that can be used to generate and modify images based on text prompts. It is a [Latent Diffusion Model](https://arxiv.org/abs/2112.10752) that uses a fixed, pretrained text encoder ([CLIP ViT-L/14](https://arxiv.org/abs/2103.00020)) as suggested in the [Imagen paper](https://arxiv.org/abs/2205.11487).46- **Resources for more information:** [GitHub Repository](https://github.com/CompVis/stable-diffusion), [Paper](https://arxiv.org/abs/2112.10752).47- **Cite as:**48 49 @InProceedings{Rombach_2022_CVPR,50 author = {Rombach, Robin and Blattmann, Andreas and Lorenz, Dominik and Esser, Patrick and Ommer, Bj\"orn},51 title = {High-Resolution Image Synthesis With Latent Diffusion Models},52 booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},53 month = {June},54 year = {2022},55 pages = {10684-10695}56 }57 58## Examples59 60We recommend using [🤗's Diffusers library](https://github.com/huggingface/diffusers) to run Stable Diffusion.61 62### PyTorch63 64```bash65pip install --upgrade diffusers transformers scipy66```67 68Running the pipeline with the default PNDM scheduler:69 70```python71import torch72from diffusers import StableDiffusionPipeline73 74model_id = "CompVis/stable-diffusion-v1-4"75device = "cuda"76 77 78pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)79pipe = pipe.to(device)80 81prompt = "a photo of an astronaut riding a horse on mars"82image = pipe(prompt).images[0] 83 84image.save("astronaut_rides_horse.png")85```86 87**Note**:88If you are limited by GPU memory and have less than 4GB of GPU RAM available, please make sure to load the StableDiffusionPipeline in float16 precision instead of the default float32 precision as done above. You can do so by telling diffusers to expect the weights to be in float16 precision:89 90 91```py92import torch93 94pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)95pipe = pipe.to(device)96pipe.enable_attention_slicing()97 98prompt = "a photo of an astronaut riding a horse on mars"99image = pipe(prompt).images[0] 100 101image.save("astronaut_rides_horse.png")102```103 104To swap out the noise scheduler, pass it to `from_pretrained`:105 106```python107from diffusers import StableDiffusionPipeline, EulerDiscreteScheduler108 109model_id = "CompVis/stable-diffusion-v1-4"110 111# Use the Euler scheduler here instead112scheduler = EulerDiscreteScheduler.from_pretrained(model_id, subfolder="scheduler")113pipe = StableDiffusionPipeline.from_pretrained(model_id, scheduler=scheduler, torch_dtype=torch.float16)114pipe = pipe.to("cuda")115 116prompt = "a photo of an astronaut riding a horse on mars"117image = pipe(prompt).images[0] 118 119image.save("astronaut_rides_horse.png")120```121 122### JAX/Flax123 124To use StableDiffusion on TPUs and GPUs for faster inference you can leverage JAX/Flax.125 126Running the pipeline with default PNDMScheduler127 128```python129import jax130import numpy as np131from flax.jax_utils import replicate132from flax.training.common_utils import shard133 134from diffusers import FlaxStableDiffusionPipeline135 136pipeline, params = FlaxStableDiffusionPipeline.from_pretrained(137 "CompVis/stable-diffusion-v1-4", revision="flax", dtype=jax.numpy.bfloat16138)139 140prompt = "a photo of an astronaut riding a horse on mars"141 142prng_seed = jax.random.PRNGKey(0)143num_inference_steps = 50144 145num_samples = jax.device_count()146prompt = num_samples * [prompt]147prompt_ids = pipeline.prepare_inputs(prompt)148 149# shard inputs and rng150params = replicate(params)151prng_seed = jax.random.split(prng_seed, num_samples)152prompt_ids = shard(prompt_ids)153 154images = pipeline(prompt_ids, params, prng_seed, num_inference_steps, jit=True).images155images = pipeline.numpy_to_pil(np.asarray(images.reshape((num_samples,) + images.shape[-3:])))156```157 158**Note**:159If you are limited by TPU memory, please make sure to load the `FlaxStableDiffusionPipeline` in `bfloat16` precision instead of the default `float32` precision as done above. You can do so by telling diffusers to load the weights from "bf16" branch.160 161```python162import jax163import numpy as np164from flax.jax_utils import replicate165from flax.training.common_utils import shard166 167from diffusers import FlaxStableDiffusionPipeline168 169pipeline, params = FlaxStableDiffusionPipeline.from_pretrained(170 "CompVis/stable-diffusion-v1-4", revision="bf16", dtype=jax.numpy.bfloat16171)172 173prompt = "a photo of an astronaut riding a horse on mars"174 175prng_seed = jax.random.PRNGKey(0)176num_inference_steps = 50177 178num_samples = jax.device_count()179prompt = num_samples * [prompt]180prompt_ids = pipeline.prepare_inputs(prompt)181 182# shard inputs and rng183params = replicate(params)184prng_seed = jax.random.split(prng_seed, num_samples)185prompt_ids = shard(prompt_ids)186 187images = pipeline(prompt_ids, params, prng_seed, num_inference_steps, jit=True).images188images = pipeline.numpy_to_pil(np.asarray(images.reshape((num_samples,) + images.shape[-3:])))189```190 191# Uses192 193## Direct Use 194The model is intended for research purposes only. Possible research areas and195tasks include196 197- Safe deployment of models which have the potential to generate harmful content.198- Probing and understanding the limitations and biases of generative models.199- Generation of artworks and use in design and other artistic processes.200- Applications in educational or creative tools.201- Research on generative models.202 203Excluded uses are described below.204 205 ### Misuse, Malicious Use, and Out-of-Scope Use206_Note: This section is taken from the [DALLE-MINI model card](https://huggingface.co/dalle-mini/dalle-mini), but applies in the same way to Stable Diffusion v1_.207 208 209The model should not be used to intentionally create or disseminate images that create hostile or alienating environments for people. This includes generating images that people would foreseeably find disturbing, distressing, or offensive; or content that propagates historical or current stereotypes.210 211#### Out-of-Scope Use212The model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model.213 214#### Misuse and Malicious Use215Using the model to generate content that is cruel to individuals is a misuse of this model. This includes, but is not limited to:216 217- Generating demeaning, dehumanizing, or otherwise harmful representations of people or their environments, cultures, religions, etc.218- Intentionally promoting or propagating discriminatory content or harmful stereotypes.219- Impersonating individuals without their consent.220- Sexual content without consent of the people who might see it.221- Mis- and disinformation222- Representations of egregious violence and gore223- Sharing of copyrighted or licensed material in violation of its terms of use.224- Sharing content that is an alteration of copyrighted or licensed material in violation of its terms of use.225 226## Limitations and Bias227 228### Limitations229 230- The model does not achieve perfect photorealism231- The model cannot render legible text232- The model does not perform well on more difficult tasks which involve compositionality, such as rendering an image corresponding to “A red cube on top of a blue sphere”233- Faces and people in general may not be generated properly.234- The model was trained mainly with English captions and will not work as well in other languages.235- The autoencoding part of the model is lossy236- The model was trained on a large-scale dataset237 [LAION-5B](https://laion.ai/blog/laion-5b/) which contains adult material238 and is not fit for product use without additional safety mechanisms and239 considerations.240- No additional measures were used to deduplicate the dataset. As a result, we observe some degree of memorization for images that are duplicated in the training data.241 The training data can be searched at [https://rom1504.github.io/clip-retrieval/](https://rom1504.github.io/clip-retrieval/) to possibly assist in the detection of memorized images.242 243### Bias244 245While the capabilities of image generation models are impressive, they can also reinforce or exacerbate social biases. 246Stable Diffusion v1 was trained on subsets of [LAION-2B(en)](https://laion.ai/blog/laion-5b/), 247which consists of images that are primarily limited to English descriptions. 248Texts and images from communities and cultures that use other languages are likely to be insufficiently accounted for. 249This affects the overall output of the model, as white and western cultures are often set as the default. Further, the 250ability of the model to generate content with non-English prompts is significantly worse than with English-language prompts.251 252### Safety Module253 254The intended use of this model is with the [Safety Checker](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/safety_checker.py) in Diffusers. 255This checker works by checking model outputs against known hard-coded NSFW concepts.256The concepts are intentionally hidden to reduce the likelihood of reverse-engineering this filter.257Specifically, the checker compares the class probability of harmful concepts in the embedding space of the `CLIPTextModel` *after generation* of the images. 258The concepts are passed into the model with the generated image and compared to a hand-engineered weight for each NSFW concept.259 260 261## Training262 263**Training Data**264The model developers used the following dataset for training the model:265 266- LAION-2B (en) and subsets thereof (see next section)267 268**Training Procedure**269Stable Diffusion v1-4 is a latent diffusion model which combines an autoencoder with a diffusion model that is trained in the latent space of the autoencoder. During training, 270 271- Images are encoded through an encoder, which turns images into latent representations. The autoencoder uses a relative downsampling factor of 8 and maps images of shape H x W x 3 to latents of shape H/f x W/f x 4272- Text prompts are encoded through a ViT-L/14 text-encoder.273- The non-pooled output of the text encoder is fed into the UNet backbone of the latent diffusion model via cross-attention.274- The loss is a reconstruction objective between the noise that was added to the latent and the prediction made by the UNet.275 276We currently provide four checkpoints, which were trained as follows.277- [`stable-diffusion-v1-1`](https://huggingface.co/CompVis/stable-diffusion-v1-1): 237,000 steps at resolution `256x256` on [laion2B-en](https://huggingface.co/datasets/laion/laion2B-en).278 194,000 steps at resolution `512x512` on [laion-high-resolution](https://huggingface.co/datasets/laion/laion-high-resolution) (170M examples from LAION-5B with resolution `>= 1024x1024`).279- [`stable-diffusion-v1-2`](https://huggingface.co/CompVis/stable-diffusion-v1-2): Resumed from `stable-diffusion-v1-1`.280 515,000 steps at resolution `512x512` on "laion-improved-aesthetics" (a subset of laion2B-en,281filtered to images with an original size `>= 512x512`, estimated aesthetics score `> 5.0`, and an estimated watermark probability `< 0.5`. The watermark estimate is from the LAION-5B metadata, the aesthetics score is estimated using an [improved aesthetics estimator](https://github.com/christophschuhmann/improved-aesthetic-predictor)).282- [`stable-diffusion-v1-3`](https://huggingface.co/CompVis/stable-diffusion-v1-3): Resumed from `stable-diffusion-v1-2`. 195,000 steps at resolution `512x512` on "laion-improved-aesthetics" and 10 % dropping of the text-conditioning to improve [classifier-free guidance sampling](https://arxiv.org/abs/2207.12598).283- [`stable-diffusion-v1-4`](https://huggingface.co/CompVis/stable-diffusion-v1-4) Resumed from `stable-diffusion-v1-2`.225,000 steps at resolution `512x512` on "laion-aesthetics v2 5+" and 10 % dropping of the text-conditioning to improve [classifier-free guidance sampling](https://arxiv.org/abs/2207.12598).284 285- **Hardware:** 32 x 8 x A100 GPUs286- **Optimizer:** AdamW287- **Gradient Accumulations**: 2288- **Batch:** 32 x 8 x 2 x 4 = 2048289- **Learning rate:** warmup to 0.0001 for 10,000 steps and then kept constant290 291## Evaluation Results 292Evaluations with different classifier-free guidance scales (1.5, 2.0, 3.0, 4.0,2935.0, 6.0, 7.0, 8.0) and 50 PLMS sampling294steps show the relative improvements of the checkpoints:295 296297 298Evaluated using 50 PLMS steps and 10000 random prompts from the COCO2017 validation set, evaluated at 512x512 resolution. Not optimized for FID scores.299## Environmental Impact300 301**Stable Diffusion v1** **Estimated Emissions**302Based on that information, we estimate the following CO2 emissions using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). The hardware, runtime, cloud provider, and compute region were utilized to estimate the carbon impact.303 304- **Hardware Type:** A100 PCIe 40GB305- **Hours used:** 150000306- **Cloud Provider:** AWS307- **Compute Region:** US-east308- **Carbon Emitted (Power consumption x Time x Carbon produced based on location of power grid):** 11250 kg CO2 eq.309 310 311## Citation312 313```bibtex314 @InProceedings{Rombach_2022_CVPR,315 author = {Rombach, Robin and Blattmann, Andreas and Lorenz, Dominik and Esser, Patrick and Ommer, Bj\"orn},316 title = {High-Resolution Image Synthesis With Latent Diffusion Models},317 booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},318 month = {June},319 year = {2022},320 pages = {10684-10695}321 }322```323 324*This model card was written by: Robin Rombach and Patrick Esser and is based on the [DALL-E Mini model card](https://huggingface.co/dalle-mini/dalle-mini).*