amd/sd1.5-controlnet-canny-amdnpu

🚀 Stable Diffusion 1.5 ControlNet (Canny) on AMD AI PC NPU

ControlNet v1.1 (Canny) adds conditional control to text-to-image diffusion models using Canny edge maps as guidance. It is designed to be used with Stable Diffusion 1.5. For more information about how ControlNet works, please see the 🧨 Diffusers ControlNet blog and the ControlNet paper.

More details about this model can be found on the original Hugging Face model card: lllyasviel/control_v11p_sd15_canny.

This model repo contains the optimized ONNX models required to run the Stable Diffusion 1.5 + ControlNet (Canny) image generation pipeline on AMD NPUs.

Model Details

The folder structure is organized to mirror the main components of the ControlNet diffusion pipeline (controlnet, scheduler, text encoder, tokenizer, UNet, VAE decoder, and supporting modules).

├─ controlnet/
├─ feature_extractor/
├─ safety_checker/
├─ scheduler/
├─ text_encoder/
├─ tokenizer/
├─ unet/
└─ vae_decoder/

The controlnet folder contains the ControlNet model conditioned on Canny edges. It is exported and structured specifically to leverage the AMD NPU accelerator together with the UNet during denoising.

The scheduler folder contains the scheduler configuration (timesteps, betas, alphas, etc.) used during the diffusion sampling process.

The text_encoder folder contains the text encoder model used to convert the input prompt into conditioning embeddings for the diffusion model.

The tokenizer contains the tokenizer configuration and vocabulary files required to preprocess the text prompt before it is fed to the text encoder.

The unet folder contains the UNet model used in the diffusion process. The UNet is exported and structured specifically to leverage the AMD NPU accelerator for the denoising steps.

The vae_decoder folder contains the VAE decoder model used to map latent representations back to the image space. The VAE decoder is also structured to make use of the NPU accelerator for efficient image reconstruction.

The feature_extractor and safety_checker folders contain supporting components used with the Stable Diffusion 1.5 pipeline.

Note: ControlNet, UNet, and VAE decoder models are optimized and structured to run on AMD NPUs. The other components (text encoder, tokenizer, scheduler, feature extractor, and safety checker) are shared between GPU and NPU pipelines, but are provided here for completeness.

⚡ Intended Use

Getting Started

To get started with this model, visit github.com/amd/sd-sandbox.

⚓ Ethical Considerations

AMD is committed to conducting our business in a fair, ethical and honest manner and in compliance with all applicable laws, rules and regulations. You can find out more at the AMD Ethics and Compliance page.

⚠️ Caveats and Recommendations

Please visit the original model card for more details on usage, preprocessing (Canny edge extraction), and limitations: lllyasviel/control_v11p_sd15_canny.

📌 Citation Details

@misc{zhang2023adding,
      title={Adding Conditional Control to Text-to-Image Diffusion Models},
      author={Lvmin Zhang and Maneesh Agrawala},
      year={2023},
      eprint={2302.05543},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}