InstantX/Qwen-Image-ControlNet-Union

1---2license: apache-2.03language:4- en5library_name: diffusers6pipeline_tag: image-to-image7tags:8- Image-to-Image9- ControlNet10- Diffusers11- QwenImageControlNetPipeline12- Qwen-Image13base_model: Qwen/Qwen-Image14---15 16# Qwen-Image-ControlNet-Union17This repository provides a unified ControlNet that supports 4 common control types (canny, soft edge, depth, pose) for [Qwen-Image](https://github.com/QwenLM/Qwen-Image).18 19 20# Model Cards21- This ControlNet consists of 5 double blocks copied from the pretrained transformer layers.22- We train the model from scratch for 50K steps using a dataset of 10M high-quality general and human images.23- We train at 1328x1328 resolution in BFloat16, batch size=64, learning rate=4e-5. We set the text drop ratio to 0.10.24- This model supports multiple control modes, including canny, soft edge, depth, pose. You can use it just as a normal ControlNet.25 26# Showcases27<table style="width:100%; table-layout:fixed;">28  <tr>29    <td><img src="./conds/canny1.png"   alt="canny"></td>30    <td><img src="./outputs/canny1.png" alt="canny"></td>31  </tr>32  <tr>33    <td><img src="./conds/soft_edge.png"    alt="soft_edge"></td>34    <td><img src="./outputs/soft_edge.png"   alt="soft_edge"></td>35  </tr>36  <tr>37    <td><img src="./conds/depth.png"    alt="depth"></td>38    <td><img src="./outputs/depth.png"   alt="depth"></td>39  </tr>40  <tr>41    <td><img src="./conds/pose.png"    alt="pose"></td>42    <td><img src="./outputs/pose.png"   alt="pose"></td>43  </tr>44</table>45 46# Inference47```python48import torch49from diffusers.utils import load_image50 51# https://github.com/huggingface/diffusers/pull/1221552# pip install git+https://github.com/huggingface/diffusers53from diffusers import QwenImageControlNetPipeline, QwenImageControlNetModel54 55base_model = "Qwen/Qwen-Image"56controlnet_model = "InstantX/Qwen-Image-ControlNet-Union"57 58controlnet = QwenImageControlNetModel.from_pretrained(controlnet_model, torch_dtype=torch.bfloat16)59 60pipe = QwenImageControlNetPipeline.from_pretrained(61    base_model, controlnet=controlnet, torch_dtype=torch.bfloat1662)63pipe.to("cuda")64 65# canny66# it is highly suggested to add 'TEXT' into prompt if there are text elements67control_image = load_image("conds/canny.png")68prompt = "Aesthetics art, traditional asian pagoda, elaborate golden accents, sky blue and white color palette, swirling cloud pattern, digital illustration, east asian architecture, ornamental rooftop, intricate detailing on building, cultural representation."69controlnet_conditioning_scale = 1.070 71# soft edge72# control_image = load_image("conds/soft_edge.png")73# prompt = "Photograph of a young man with light brown hair jumping mid-air off a large, reddish-brown rock. He's wearing a navy blue sweater, light blue shirt, gray pants, and brown shoes. His arms are outstretched, and he has a slight smile on his face. The background features a cloudy sky and a distant, leafless tree line. The grass around the rock is patchy."74# controlnet_conditioning_scale = 1.075 76# depth77# control_image = load_image("conds/depth.png")78# prompt = "A swanky, minimalist living room with a huge floor-to-ceiling window letting in loads of natural light. A beige couch with white cushions sits on a wooden floor, with a matching coffee table in front. The walls are a soft, warm beige, decorated with two framed botanical prints. A potted plant chills in the corner near the window. Sunlight pours through the leaves outside, casting cool shadows on the floor."79# controlnet_conditioning_scale = 1.080 81# pose82# control_image = load_image("conds/pose.png")83# prompt = "Photograph of a young man with light brown hair and a beard, wearing a beige flat cap, black leather jacket, gray shirt, brown pants, and white sneakers. He's sitting on a concrete ledge in front of a large circular window, with a cityscape reflected in the glass. The wall is cream-colored, and the sky is clear blue. His shadow is cast on the wall."84# controlnet_conditioning_scale = 1.085 86image = pipe(87    prompt=prompt,88    negative_prompt=" ",89    control_image=control_image,90    controlnet_conditioning_scale=controlnet_conditioning_scale,91    width=control_image.size[0],92    height=control_image.size[1],93    num_inference_steps=30,94    true_cfg_scale=4.0,95    generator=torch.Generator(device="cuda").manual_seed(42),96).images[0]97image.save(f"qwenimage_cn_union_result.png")98```99 100# Inference Setting101You can adjust control strength via controlnet_conditioning_scale.102- Canny: use cv2.Canny, set controlnet_conditioning_scale in [0.8, 1.0]103- Soft Edge: use [AnylineDetector](https://github.com/huggingface/controlnet_aux), set controlnet_conditioning_scale in [0.8, 1.0]104- Depth: use [depth-anything](https://github.com/DepthAnything/Depth-Anything-V2), set controlnet_conditioning_scale in [0.8, 1.0]105- Pose: use [DWPose](https://github.com/IDEA-Research/DWPose/tree/onnx), set controlnet_conditioning_scale in [0.8, 1.0]106 107We strongly recommend using detailed prompts, especially when include text elements. For example, use "a poster with text 'InstantX Team' on the top" instead of "a poster".108 109For multiple conditions inference, please refer to [PR](https://github.com/huggingface/diffusers/pull/12215).110 111# ComfyUI Support112[ComfyUI](https://www.comfy.org/) offers native support for Qwen-Image-ControlNet-Union. Check the [blog](https://blog.comfy.org/p/day-1-support-of-qwen-image-instantx) for more details.113 114# Community Support115[Liblib AI](https://www.liblib.art/) offers native support for Qwen-Image-ControlNet-Union. [Visit](https://www.liblib.art/sd) for online inference.116 117# Limitations118We find that the model was unable to preserve some details without explicit 'TEXT' in prompt, such as small font text.119 120# Acknowledgements121This model is developed by InstantX Team. All copyright reserved.122