Runware/FLUX.1-dev-ControlNet-Union-Pro-2.0

1---2license: other3license_name: flux-1-dev-non-commercial-license4license_link: https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md5 6language:7  - en8library_name: diffusers9pipeline_tag: text-to-image10 11tags:12- Text-to-Image13- ControlNet14- Diffusers15- Flux.1-dev16- image-generation17- Stable Diffusion18base_model: black-forest-labs/FLUX.1-dev19---20 21# FLUX.1-dev-ControlNet-Union-Pro-2.022 23This repository contains an unified ControlNet for FLUX.1-dev model released by [Shakker Labs](https://huggingface.co/Shakker-Labs). We provide an [online demo](https://huggingface.co/spaces/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0). A FP8 quantized version provided by community can be found in [ABDALLALSWAITI/FLUX.1-dev-ControlNet-Union-Pro-2.0-fp8](https://huggingface.co/ABDALLALSWAITI/FLUX.1-dev-ControlNet-Union-Pro-2.0-fp8).24 25# Keynotes26In comparison with [Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro](https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro),27- Remove mode embedding, has smaller model size.28- Improve on canny and pose, better control and aesthetics.29- Add support for soft edge. Remove support for tile.30 31# Model Cards32- This ControlNet consists of 6 double blocks and 0 single block. Mode embedding is removed.33- We train the model from scratch for 300k steps using a dataset of 20M high-quality general and human images. We train at 512x512 resolution in BFloat16, batch size = 128, learning rate = 2e-5, the guidance is uniformly sampled from [1, 7]. We set the text drop ratio to 0.20.34- This model supports multiple control modes, including canny, soft edge, depth, pose, gray. You can use it just as a normal ControlNet.35- This model can be jointly used with other ControlNets.36 37# Showcases38 39<table>40  <tr>41    <td><img src="./images/canny.png" alt="canny" style="height:100%"></td>42  </tr>43  <tr>44    <td><img src="./images/softedge.png" alt="softedge" style="height:100%"></td>45  </tr>46  <tr>47    <td><img src="./images/pose.png" alt="pose" style="height:100%"></td>48  </tr>49  <tr>50    <td><img src="./images/depth.png" alt="depth" style="height:100%"></td>51  </tr>52  <tr>53    <td><img src="./images/gray.png" alt="gray" style="height:100%"></td>54  </tr>55</table>56 57# Inference58```python59import torch60from diffusers.utils import load_image61from diffusers import FluxControlNetPipeline, FluxControlNetModel62 63base_model = 'black-forest-labs/FLUX.1-dev'64controlnet_model_union = 'Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0'65 66controlnet = FluxControlNetModel.from_pretrained(controlnet_model_union, torch_dtype=torch.bfloat16)67pipe = FluxControlNetPipeline.from_pretrained(base_model, controlnet=controlnet, torch_dtype=torch.bfloat16)68pipe.to("cuda")69 70# replace with other conds71control_image = load_image("./conds/canny.png")72width, height = control_image.size73 74prompt = "A young girl stands gracefully at the edge of a serene beach, her long, flowing hair gently tousled by the sea breeze. She wears a soft, pastel-colored dress that complements the tranquil blues and greens of the coastal scenery. The golden hues of the setting sun cast a warm glow on her face, highlighting her serene expression. The background features a vast, azure ocean with gentle waves lapping at the shore, surrounded by distant cliffs and a clear, cloudless sky. The composition emphasizes the girl's serene presence amidst the natural beauty, with a balanced blend of warm and cool tones."75 76image = pipe(77    prompt, 78    control_image=control_image,79    width=width,80    height=height,81    controlnet_conditioning_scale=0.7,82    control_guidance_end=0.8,83    num_inference_steps=30, 84    guidance_scale=3.5,85    generator=torch.Generator(device="cuda").manual_seed(42),86).images[0]87```88 89# Multi-Inference90```python91import torch92from diffusers.utils import load_image93 94# https://github.com/huggingface/diffusers/pull/1135095# You can directly import from diffusers by install the laster version from source96# from diffusers import FluxControlNetPipeline, FluxControlNetModel97 98# use local files for this moment99from pipeline_flux_controlnet import FluxControlNetPipeline100from controlnet_flux import FluxControlNetModel101 102base_model = 'black-forest-labs/FLUX.1-dev'103controlnet_model_union = 'Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0'104 105controlnet = FluxControlNetModel.from_pretrained(controlnet_model_union, torch_dtype=torch.bfloat16)106pipe = FluxControlNetPipeline.from_pretrained(base_model, controlnet=[controlnet], torch_dtype=torch.bfloat16) # use [] to enable multi-CNs107pipe.to("cuda")108 109# replace with other conds110control_image = load_image("./conds/canny.png")111width, height = control_image.size112 113prompt = "A young girl stands gracefully at the edge of a serene beach, her long, flowing hair gently tousled by the sea breeze. She wears a soft, pastel-colored dress that complements the tranquil blues and greens of the coastal scenery. The golden hues of the setting sun cast a warm glow on her face, highlighting her serene expression. The background features a vast, azure ocean with gentle waves lapping at the shore, surrounded by distant cliffs and a clear, cloudless sky. The composition emphasizes the girl's serene presence amidst the natural beauty, with a balanced blend of warm and cool tones."114 115image = pipe(116    prompt, 117    control_image=[control_image, control_image], # try with different conds such as canny&depth, pose&depth118    width=width,119    height=height,120    controlnet_conditioning_scale=[0.35, 0.35],121    control_guidance_end=[0.8, 0.8],122    num_inference_steps=30, 123    guidance_scale=3.5,124    generator=torch.Generator(device="cuda").manual_seed(42),125).images[0]126```127 128# Recommended Parameters129You can adjust controlnet_conditioning_scale and control_guidance_end for stronger control and better detail preservation. For better stability, we highly suggest to use detailed prompt, for some cases, multi-conditions help.130- Canny: use cv2.Canny, controlnet_conditioning_scale=0.7, control_guidance_end=0.8.131- Soft Edge: use [AnylineDetector](https://github.com/huggingface/controlnet_aux), controlnet_conditioning_scale=0.7, control_guidance_end=0.8.132- Depth: use [depth-anything](https://github.com/DepthAnything/Depth-Anything-V2), controlnet_conditioning_scale=0.8, control_guidance_end=0.8.133- Pose: use [DWPose](https://github.com/IDEA-Research/DWPose/tree/onnx), controlnet_conditioning_scale=0.9, control_guidance_end=0.65.134- Gray: use cv2.cvtColor, controlnet_conditioning_scale=0.9, control_guidance_end=0.8.135 136# Resources137- [InstantX/FLUX.1-dev-IP-Adapter](https://huggingface.co/InstantX/FLUX.1-dev-IP-Adapter)138- [InstantX/FLUX.1-dev-Controlnet-Canny](https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Canny)139- [Shakker-Labs/FLUX.1-dev-ControlNet-Depth](https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Depth)140- [Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro](https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro)141 142# Acknowledgements143This model is developed by [Shakker Labs](https://huggingface.co/Shakker-Labs). The original idea is inspired by [xinsir/controlnet-union-sdxl-1.0](https://huggingface.co/xinsir/controlnet-union-sdxl-1.0). All copyright reserved.144 145# Citation146If you find this project useful in your research, please cite us via147```148@misc{flux-cn-union-pro-2,149    author = {Shakker-Labs},150    title = {ControlNet-Union},151    year = {2025},152    howpublished={\url{https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0}},153}154```