zeromodels/stable-diffusion-3.5-large
125
1---2pipeline_tag: text-to-image3license: other4license_name: stabilityai-ai-community5license_link: https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md6base_model: stabilityai/stable-diffusion-3.5-large7library_name: zeromodels8language:9- en10tags:11- keras12- zeromodels13- stable-diffusion14- stable-diffusion-315- sd316- mmdit17- text-to-image18- diffusion19- rectified-flow20- arxiv:2403.0320621- pytorch22- jax23- tf24---25*See [our collection](https://huggingface.co/collections/zeromodels/stable-diffusion-v35-6aa7963880e76656e5646ffc) for all Stable Diffusion 3.5 checkpoints.*26 27# Run Stable Diffusion 3.5 with Keras 3: JAX, PyTorch, or TensorFlow28 29[](https://github.com/IMvision12/ZeroModels) [](https://imvision12.github.io/ZeroModels/stable_diffusion_3_5/) [](https://huggingface.co/collections/zeromodels/stable-diffusion-v35-6aa7963880e76656e5646ffc)30 31**Powered by Stability AI**32 33# zeromodels/stable-diffusion-3.5-large34 35Paper: [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (arXiv:2403.03206)](https://arxiv.org/abs/2403.03206) | [HF Papers](https://huggingface.co/papers/2403.03206)36 37Pure-**Keras 3** conversion of [`stabilityai/stable-diffusion-3.5-large`](https://huggingface.co/stabilityai/stable-diffusion-3.5-large) for38[zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on39**TensorFlow / Torch / JAX**. The container ships the MMDiT denoiser, the 16-channel VAE40and the two CLIP text encoders in `model.weights.json` shards (9.05B parameters,4117.01 GB), plus `zm_config.json` (the component configs, the checkpoint's42`FlowMatchEulerDiscreteScheduler` (shift 3.0) and the default43generation settings) and the two tokenizers (`tokenizer.json`, the CLIP BPE, and44`tokenizer_3.json`, the T5 SentencePiece). Weights are stored in **float16**, the45checkpoint's native precision (the VAE in float32), and load in float16 by default; pass46`load_dtype="float32"` to `from_weights` for a float32 model. This checkpoint generates47**1024x1024** images (a 128x128x16 latent).48 49The third text encoder, the 4.7B-parameter **T5-XXL**, is shared by every SD 3 / 3.550checkpoint and hosted once at51[`zeromodels/t5-v1_1-xxl-encoder`](https://huggingface.co/zeromodels/t5-v1_1-xxl-encoder); attach it52with `text_encoder_3=` (below) or leave it out (the T5 features are zeroed, SD 3's53memory-saving mode).54 55For model details, intended use and limitations, see the upstream56[model card](https://huggingface.co/stabilityai/stable-diffusion-3.5-large).57 58## Architecture59 60| Component | zeromodels class | Details |61| --- | --- | --- |62| Denoiser | `SD3Transformer2DModel` | MMDiT: 38 joint blocks of 38 x 64-d heads (2432 wide), patch 2, 4096-d text features projected to 2432, 2048-d pooled conditioning, 192x192 position grid, RMS-normalized queries and keys |63| Autoencoder | `AutoencoderKL` | (128, 256, 512, 512) channels, x8 spatial compression to 16 latent channels, `scaling_factor` 1.5305, `shift_factor` 0.0609, float32 (`force_upcast`) |64| Text encoder | functional CLIP text tower | CLIP ViT-L/14: 768-d, 12 layers, `quick_gelu`, 768-d projection; penultimate hidden state + projected pooled state |65| Text encoder 2 | functional CLIP text tower | OpenCLIP ViT-bigG/14: 1280-d, 32 layers, `gelu`, 1280-d projection; penultimate hidden state + projected pooled state |66| Text encoder 3 | `SD3T5EncoderModel` (separate repo) | T5 v1.1 XXL encoder: 4096-d, 24 layers, gated GELU, 256 tokens |67| Scheduler | `FlowMatchEulerDiscreteScheduler` | rectified flow over 1000 timesteps, shift 3.0 |68 69## Quick start70 71```python72import os73os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"74 75from PIL import Image76from zeromodels.models.stable_diffusion_3_5 import StableDiffusion3_5TextToImage, StableDiffusion3_5Tokenizer77 78model = StableDiffusion3_5TextToImage.from_weights(79 "zeromodels/stable-diffusion-3.5-large",80 text_encoder_3="zeromodels/t5-v1_1-xxl-encoder", # optional: omit to zero the T5 features81)82tokenizer = StableDiffusion3_5Tokenizer.from_weights("zeromodels/stable-diffusion-3.5-large")83 84inputs = tokenizer("a photograph of an astronaut riding a horse")85images = model.generate(**inputs, num_inference_steps=28, guidance_scale=3.5, seed=0)86Image.fromarray(images[0]).save("astronaut.png") # (1024, 1024, 3) uint887```88 89`generate` takes the tokenizer's `input_ids` / `attention_mask` / `input_ids_3` (batch them90for several prompts), an optional tokenized negative prompt (`negative_input_ids` /91`negative_input_ids_3`), `num_inference_steps`, `guidance_scale`, a `seed`, or explicit92`latents` of shape `(batch, 128, 128, 16)` for results that are identical across93backends; `image` / `strength` refine an image instead.94 95Load any Stable Diffusion 3.5 checkpoint the same way with `from_weights("zeromodels/<variant>")`:96 97| Variant | Hub | Training |98| --- | --- | --- |99| `stable-diffusion-3.5-large` | [zeromodels/stable-diffusion-3.5-large](https://huggingface.co/zeromodels/stable-diffusion-3.5-large) | 1024px, rectified flow (shift 3), 28 steps at guidance 3.5: the 8B MMDiT with RMS-normalized queries and keys |100| `stable-diffusion-3.5-large-turbo` | [zeromodels/stable-diffusion-3.5-large-turbo](https://huggingface.co/zeromodels/stable-diffusion-3.5-large-turbo) | 1024px, 4 steps, no guidance: SD 3.5 large distilled with Adversarial Diffusion Distillation |101| `stable-diffusion-3.5-medium` | [zeromodels/stable-diffusion-3.5-medium](https://huggingface.co/zeromodels/stable-diffusion-3.5-medium) | 1024px (up to 2 MP), rectified flow (shift 3), 40 steps at guidance 4.5: the 2.5B MMDiT-X with dual-attention blocks |102 103## Tips104 105- Set `KERAS_BACKEND` **before** importing Keras / zeromodels.106- The graphs are built for 1024px. Pass `transformer_sample_size=<px / 8>, vae_sample_size=<px>` to107 `from_weights` to build for another multiple of 16px (the weights are resolution-independent108 up to the 3072px position grid).109- `StableDiffusion3_5Model.from_weights(...)` loads the same repo as the bare container110 (`.transformer` / `.vae` / `.text_encoder` / `.text_encoder_2`) without the generation loop.111- `model.text_encoder_3` can be any `SD3T5EncoderModel` (for example one loaded with112 `quantization="int8"`); it is not part of the container's weights.113- Both `channels_last` and `channels_first` are supported (`keras.config.set_image_data_format`114 before loading); `generate` always returns `(batch, H, W, 3)` uint8.115- On-the-fly `hf:` conversion is not supported for diffusion models; the checkpoints are116 hosted here, converted once.117- See the [Stable Diffusion 3.5 docs](https://imvision12.github.io/ZeroModels/stable_diffusion_3_5/).118 119## License120 121The weights are redistributed under the [Stability AI Community License](https://huggingface.co/stabilityai/stable-diffusion-3.5-large/blob/main/LICENSE.md) of the upstream122checkpoint, including its use-based restrictions. By using them you agree to those terms.123 124## Notice125 126This Stability AI Model is licensed under the Stability AI Community License, Copyright © Stability AI Ltd. All Rights Reserved127 128Modifications by zeromodels (https://github.com/IMvision12/ZeroModels): the checkpoint129released at https://huggingface.co/stabilityai/stable-diffusion-3.5-large was converted to130the Keras 3 weights layout of zeromodels (`model.weights.json, model_00000.weights.h5,131model_00001.weights.h5, model_00002.weights.h5, model_00003.weights.h5, zm_config.json,132tokenizer.json, tokenizer_3.json`), stored in float16, the upstream fp16 files, with the VAE133in float32. The model architecture and the parameter values are unchanged; the weight names134and the file format differ from the release.135 136**Powered by Stability AI**137 138## Special Thanks139 140Thank you to Stability AI for training and releasing Stable Diffusion, and to the141Hugging Face diffusers team, whose implementation this port was verified against.142