keras/stable_diffusion_3.5_large
012
1---2library_name: keras-hub3tags:4- text-to-image5pipeline_tag: text-to-image6---7### Model Overview8[Stable Diffusion 3.5 ](https://stability.ai/learning-hub/stable-diffusion-3-5-prompt-guide) is a Multimodal Diffusion Transformer (MMDiT) text-to-image model that features greatly improved performance in image quality, typography, complex prompt understanding, and resource-efficiency.9 10For more technical details, please refer to the [Research paper](https://stability.ai/news/stable-diffusion-3-research-paper).11 12Please note: this model is released under the Stability Community License. For Enterprise License visit Stability.ai or [contact us](https://stability.ai/enterprise) for commercial licensing details.13 14## Links15 16* [SD3.5 Quickstart Notebook ](https://colab.sandbox.google.com/gist/laxmareddyp/55daf77f87730c3b3f498318672f70b3/stablediffusion3_5-quckstart-notebook.ipynb)17* [SD3.5 API Documentation](https://keras.io/keras_hub/api/models/stable_diffusion_3/)18* [SD3.5 Model Card](https://huggingface.co/stabilityai/stable-diffusion-3.5-large)19* [KerasHub Beginner Guide](https://keras.io/guides/keras_hub/getting_started/)20* [KerasHub Model Publishing Guide](https://keras.io/guides/keras_hub/upload/)21 22## Presets23 24The following model checkpoints are provided by the Keras team. Full code examples for each are available below.25| Preset name | Parameters | Description |26|----------------|------------|--------------------------------------------------|27| stable_diffusion_3.5_large| 9.05B | 9 billion parameter, including CLIP L and CLIP G text encoders, MMDiT generative model, and VAE autoencoder. Developed by Stability AI.|28| stable_diffusion_3.5_large_turbo | 9.05B | 9 billion parameter, including CLIP L and CLIP G text encoders, MMDiT generative model, and VAE autoencoder. A timestep-distilled version that eliminates classifier-free guidance and uses fewer steps for generation. Developed by Stability AI. |29 30### Model Description31 32- **Developed by:** Stability AI33- **Model type:** MMDiT text-to-image generative model34- **Model Description:** This is a model that can be used to generate images based on text prompts. It is a [Multimodal Diffusion Transformer](https://arxiv.org/abs/2403.03206)35 that uses three fixed, pretrained text encoders (OpenCLIP-ViT/G, CLIP-ViT/L and T5-xxl), and QK-normalization to improve training stability.36 37## Example Usage38```python39!pip install -U keras-hub40!pip install -U keras41```42 43```44# Pretrained Stable Diffusion 3 model.45model = keras_hub.models.StableDiffusion3Backbone.from_preset(46 "stable_diffusion_3.5_large"47)48 49# Randomly initialized Stable Diffusion 3 model with custom config.50vae = keras_hub.models.VAEBackbone(...)51clip_l = keras_hub.models.CLIPTextEncoder(...)52clip_g = keras_hub.models.CLIPTextEncoder(...)53model = keras_hub.models.StableDiffusion3Backbone(54 mmdit_patch_size=2,55 mmdit_num_heads=4,56 mmdit_hidden_dim=256,57 mmdit_depth=4,58 mmdit_position_size=192,59 vae=vae,60 clip_l=clip_l,61 clip_g=clip_g,62)63 64# Image to image example65image_to_image = keras_hub.models.StableDiffusion3ImageToImage.from_preset(66 "stable_diffusion_3.5_large", height=512, width=51267)68image_to_image.generate(69 {70 "images": np.ones((512, 512, 3), dtype="float32"),71 "prompts": "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k",72 }73)74 75# Generate with batched prompts.76image_to_image.generate(77 {78 "images": np.ones((2, 512, 512, 3), dtype="float32"),79 "prompts": ["cute wallpaper art of a cat", "cute wallpaper art of a dog"],80 }81)82 83# Generate with different `num_steps`, `guidance_scale` and `strength`.84image_to_image.generate(85 {86 "images": np.ones((512, 512, 3), dtype="float32"),87 "prompts": "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k",88 }89 num_steps=50,90 guidance_scale=5.0,91 strength=0.6,92)93 94# Generate with `negative_prompts`.95text_to_image.generate(96 {97 "images": np.ones((512, 512, 3), dtype="float32"),98 "prompts": "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k",99 "negative_prompts": "green color",100 }101)102 103# inpainting example104reference_image = np.ones((1024, 1024, 3), dtype="float32")105reference_mask = np.ones((1024, 1024), dtype="float32")106inpaint = keras_hub.models.StableDiffusion3Inpaint.from_preset(107 "stable_diffusion_3.5_large", height=512, width=512108)109inpaint.generate(110 reference_image,111 reference_mask,112 "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k",113)114 115# Generate with batched prompts.116reference_images = np.ones((2, 512, 512, 3), dtype="float32")117reference_mask = np.ones((2, 1024, 1024), dtype="float32")118inpaint.generate(119 reference_images,120 reference_mask,121 ["cute wallpaper art of a cat", "cute wallpaper art of a dog"]122)123 124# Generate with different `num_steps`, `guidance_scale` and `strength`.125inpaint.generate(126 reference_image,127 reference_mask,128 "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k",129 num_steps=50,130 guidance_scale=5.0,131 strength=0.6,132)133 134# text to image example135text_to_image = keras_hub.models.StableDiffusion3TextToImage.from_preset(136 "stable_diffusion_3.5_large", height=512, width=512137)138text_to_image.generate(139 "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k"140)141 142# Generate with batched prompts.143text_to_image.generate(144 ["cute wallpaper art of a cat", "cute wallpaper art of a dog"]145)146 147# Generate with different `num_steps` and `guidance_scale`.148text_to_image.generate(149 "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k",150 num_steps=50,151 guidance_scale=5.0,152)153 154# Generate with `negative_prompts`.155text_to_image.generate(156 {157 "prompts": "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k",158 "negative_prompts": "green color",159 }160)161```162 163## Example Usage with Hugging Face URI164 165```python166!pip install -U keras-hub167!pip install -U keras168```169 170```171# Pretrained Stable Diffusion 3 model.172model = keras_hub.models.StableDiffusion3Backbone.from_preset(173 "hf://keras/stable_diffusion_3.5_large"174)175 176# Randomly initialized Stable Diffusion 3 model with custom config.177vae = keras_hub.models.VAEBackbone(...)178clip_l = keras_hub.models.CLIPTextEncoder(...)179clip_g = keras_hub.models.CLIPTextEncoder(...)180model = keras_hub.models.StableDiffusion3Backbone(181 mmdit_patch_size=2,182 mmdit_num_heads=4,183 mmdit_hidden_dim=256,184 mmdit_depth=4,185 mmdit_position_size=192,186 vae=vae,187 clip_l=clip_l,188 clip_g=clip_g,189)190 191# Image to image example192image_to_image = keras_hub.models.StableDiffusion3ImageToImage.from_preset(193 "hf://keras/stable_diffusion_3.5_large", height=512, width=512194)195image_to_image.generate(196 {197 "images": np.ones((512, 512, 3), dtype="float32"),198 "prompts": "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k",199 }200)201 202# Generate with batched prompts.203image_to_image.generate(204 {205 "images": np.ones((2, 512, 512, 3), dtype="float32"),206 "prompts": ["cute wallpaper art of a cat", "cute wallpaper art of a dog"],207 }208)209 210# Generate with different `num_steps`, `guidance_scale` and `strength`.211image_to_image.generate(212 {213 "images": np.ones((512, 512, 3), dtype="float32"),214 "prompts": "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k",215 }216 num_steps=50,217 guidance_scale=5.0,218 strength=0.6,219)220 221# Generate with `negative_prompts`.222text_to_image.generate(223 {224 "images": np.ones((512, 512, 3), dtype="float32"),225 "prompts": "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k",226 "negative_prompts": "green color",227 }228)229 230# inpainting example231reference_image = np.ones((1024, 1024, 3), dtype="float32")232reference_mask = np.ones((1024, 1024), dtype="float32")233inpaint = keras_hub.models.StableDiffusion3Inpaint.from_preset(234 "hf://keras/stable_diffusion_3.5_large", height=512, width=512235)236inpaint.generate(237 reference_image,238 reference_mask,239 "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k",240)241 242# Generate with batched prompts.243reference_images = np.ones((2, 512, 512, 3), dtype="float32")244reference_mask = np.ones((2, 1024, 1024), dtype="float32")245inpaint.generate(246 reference_images,247 reference_mask,248 ["cute wallpaper art of a cat", "cute wallpaper art of a dog"]249)250 251# Generate with different `num_steps`, `guidance_scale` and `strength`.252inpaint.generate(253 reference_image,254 reference_mask,255 "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k",256 num_steps=50,257 guidance_scale=5.0,258 strength=0.6,259)260 261# text to image example262text_to_image = keras_hub.models.StableDiffusion3TextToImage.from_preset(263 "hf://keras/stable_diffusion_3.5_large", height=512, width=512264)265text_to_image.generate(266 "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k"267)268 269# Generate with batched prompts.270text_to_image.generate(271 ["cute wallpaper art of a cat", "cute wallpaper art of a dog"]272)273 274# Generate with different `num_steps` and `guidance_scale`.275text_to_image.generate(276 "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k",277 num_steps=50,278 guidance_scale=5.0,279)280 281# Generate with `negative_prompts`.282text_to_image.generate(283 {284 "prompts": "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k",285 "negative_prompts": "green color",286 }287)288```289