zeromodels/stable-diffusion-v1-4
0101
1---2pipeline_tag: text-to-image3license: creativeml-openrail-m4base_model: CompVis/stable-diffusion-v1-45library_name: zeromodels6language:7- en8tags:9- keras10- zeromodels11- stable-diffusion12- stable-diffusion-diffusers13- text-to-image14- diffusion15- latent-diffusion16- arxiv:2112.1075217- pytorch18- jax19- tf20---21*See [our collection](https://huggingface.co/zeromodels) for all Stable Diffusion 1.x checkpoints.*22 23# Run Stable Diffusion 1.x with Keras 3: JAX, PyTorch, or TensorFlow24 25[](https://github.com/IMvision12/ZeroModels) [](https://imvision12.github.io/ZeroModels/stable_diffusion/) [](https://huggingface.co/zeromodels)26 27# zeromodels/stable-diffusion-v1-428 29Paper: [High-Resolution Image Synthesis with Latent Diffusion Models (arXiv:2112.10752)](https://arxiv.org/abs/2112.10752) | [HF Papers](https://huggingface.co/papers/2112.10752)30 31Pure-**Keras 3** conversion of [`CompVis/stable-diffusion-v1-4`](https://huggingface.co/CompVis/stable-diffusion-v1-4) for32[zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on33**TensorFlow / Torch / JAX**. The whole text-to-image model ships as **one container**:34the UNet denoiser, the VAE and the CLIP ViT-L/14 text encoder in `model.weights.h5`35(1.07B parameters, 3.97 GB), plus `zm_config.json` (the three36component configs, the checkpoint's `PNDMScheduler` schedule with its `epsilon` objective37and the default generation settings) and the tokenizer as `tokenizer.json`. Weights are stored in **float32**, exactly as released.38This checkpoint generates **512x512** images (a 64x64 latent).39 40For model details, intended use and limitations, see the upstream41[model card](https://huggingface.co/CompVis/stable-diffusion-v1-4).42 43## Architecture44 45| Component | zeromodels class | Details |46| --- | --- | --- |47| Denoiser | `UNet2DConditionModel` | (320, 640, 1280, 1280) channels, 2 ResNet blocks per level, 8 attention heads on the 768-d text context, 1x1 conv token projection, 64x64x4 latent |48| Autoencoder | `AutoencoderKL` | (128, 256, 512, 512) channels, x8 spatial compression to 4 latent channels, `scaling_factor` 0.18215 |49| Text encoder | `CLIPTextModel` | CLIP ViT-L/14 text encoder: 768-d, 12 layers, 12 heads, 77 tokens, `quick_gelu` |50| Scheduler | `PNDMScheduler` | scaled_linear betas 0.00085 to 0.012 over 1000 steps, `epsilon`; DDIM / PNDM / Euler / Euler-ancestral are drop-in |51 52## Quick start53 54```python55import os56os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"57 58from PIL import Image59from zeromodels.models.stable_diffusion import StableDiffusionTextToImage, StableDiffusionTokenizer60 61model = StableDiffusionTextToImage.from_weights("zeromodels/stable-diffusion-v1-4")62tokenizer = StableDiffusionTokenizer.from_weights("zeromodels/stable-diffusion-v1-4")63 64inputs = tokenizer("a photograph of an astronaut riding a horse")65images = model.generate(**inputs, num_inference_steps=50, guidance_scale=7.5, seed=0)66Image.fromarray(images[0]).save("astronaut.png") # (512, 512, 3) uint867```68 69`generate` takes the tokenizer's `input_ids` (batch them for several prompts), an optional70`negative_input_ids` (tokenize the negative prompt), `num_inference_steps`, `guidance_scale`,71a `seed`, or explicit `latents` of shape `(batch, 64, 64, 4)` for results that are72identical across backends.73 74Load any Stable Diffusion 1.x checkpoint the same way with `from_weights("zeromodels/<variant>")`:75 76| Variant | Hub | Training |77| --- | --- | --- |78| `stable-diffusion-v1-1` | [zeromodels/stable-diffusion-v1-1](https://huggingface.co/zeromodels/stable-diffusion-v1-1) | 237k steps at 256px on laion2B-en, then 194k steps at 512px on laion-high-resolution |79| `stable-diffusion-v1-2` | [zeromodels/stable-diffusion-v1-2](https://huggingface.co/zeromodels/stable-diffusion-v1-2) | v1-1 + 515k steps at 512px on laion-aesthetics v2 5+ |80| `stable-diffusion-v1-3` | [zeromodels/stable-diffusion-v1-3](https://huggingface.co/zeromodels/stable-diffusion-v1-3) | v1-2 + 195k steps at 512px, 10% text-conditioning dropout (classifier-free guidance) |81| `stable-diffusion-v1-4` | [zeromodels/stable-diffusion-v1-4](https://huggingface.co/zeromodels/stable-diffusion-v1-4) | v1-2 + 225k steps at 512px, 10% text-conditioning dropout (classifier-free guidance) |82| `stable-diffusion-v1-5` | [zeromodels/stable-diffusion-v1-5](https://huggingface.co/zeromodels/stable-diffusion-v1-5) | v1-2 + 595k steps at 512px, 10% text-conditioning dropout (classifier-free guidance) |83 84## Tips85 86- Set `KERAS_BACKEND` **before** importing Keras / zeromodels.87- The graphs are built for 512px. Pass `unet_sample_size=<px / 8>, vae_sample_size=<px>` to88 `from_weights` to build for another multiple of 64px (the weights are resolution-independent).89- Swap the sampler any time: `model.scheduler = EulerDiscreteScheduler.from_config(model.config.scheduler_config)`90 (`zeromodels.base.base_scheduler`).91- `StableDiffusionModel.from_weights(...)` loads the same repo as the bare container92 (UNet / VAE / text encoder as `.unet` / `.vae` / `.text_encoder`) without the generation loop.93 94- Both `channels_last` and `channels_first` are supported (`keras.config.set_image_data_format`95 before loading); `generate` always returns `(batch, H, W, 3)` uint8.96- On-the-fly `hf:` conversion is not supported for diffusion models; the checkpoints are97 hosted here, converted once.98- See the [Stable Diffusion 1.x docs](https://imvision12.github.io/ZeroModels/stable_diffusion/).99 100## License101 102The weights are redistributed under the [CreativeML OpenRAIL-M License](https://huggingface.co/spaces/CompVis/stable-diffusion-license/blob/main/license.txt) of the upstream103checkpoint, including its use-based restrictions. By using them you agree to those terms.104 105## Notice106 107Modifications by zeromodels (https://github.com/IMvision12/ZeroModels): the checkpoint108released at https://huggingface.co/CompVis/stable-diffusion-v1-4 was converted to the Keras1093 weights layout of zeromodels (`model.weights.h5, zm_config.json, tokenizer.json`), stored110in float32 as released. The model architecture and the parameter values are unchanged; the111weight names and the file format differ from the release.112 113## Special Thanks114 115Thank you to the CompVis group at LMU Munich, Runway and Stability AI for training and releasing Stable Diffusion, and to the116Hugging Face diffusers team, whose implementation this port was verified against.117