Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

Support ControlNet for Qwen-Image #12215

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yiyixuxu merged 16 commits into huggingface:main from haofanwang:qwen-image-cn-union
Aug 22, 2025

Conversation

Copy link
Contributor

@haofanwang haofanwang commented Aug 22, 2025 •
edited
Loading

What does this PR do?

Add ControlNet-Union (InstantX/Qwen-Image-ControlNet-Union) support for Qwen-Image.

Inference

import torch
from diffusers.utils import load_image
from diffusers import QwenImageControlNetPipeline, QwenImageControlNetModel
base_model = "Qwen/Qwen-Image"
controlnet_model = "InstantX/Qwen-Image-ControlNet-Union"
controlnet = QwenImageControlNetModel.from_pretrained(controlnet_model, torch_dtype=torch.bfloat16)
pipe = QwenImageControlNetPipeline.from_pretrained(
 base_model, controlnet=controlnet, torch_dtype=torch.bfloat16
)
pipe.to("cuda")
# canny
control_image = load_image("https://huggingface.co/InstantX/Qwen-Image-ControlNet-Union/resolve/main/conds/canny.png")
prompt = "Aesthetics art, traditional asian pagoda, elaborate golden accents, sky blue and white color palette, swirling cloud pattern, digital illustration, east asian architecture, ornamental rooftop, intricate detailing on building, cultural representation."
controlnet_conditioning_scale = 1.0
image = pipe(
 prompt=prompt,
 negative_prompt=" ",
 control_image=control_image,
 controlnet_conditioning_scale=controlnet_conditioning_scale,
 width=control_image.size[0],
 height=control_image.size[1],
 num_inference_steps=30,
 true_cfg_scale=4.0,
 generator=torch.Generator(device="cuda").manual_seed(42),
).images[0]
image.save(f"qwenimage_cn_union_result.png")

Multi-Conditions

import torch
from diffusers.utils import load_image
from diffusers import QwenImageControlNetPipeline, QwenImageControlNetModel, QwenImageMultiControlNetModel
base_model = "Qwen/Qwen-Image"
controlnet_model = "InstantX/Qwen-Image-ControlNet-Union"
controlnet = QwenImageControlNetModel.from_pretrained(controlnet_model, torch_dtype=torch.bfloat16)
controlnet = QwenImageMultiControlNetModel([controlnet])
pipe = QwenImageControlNetPipeline.from_pretrained(
 base_model, controlnet=controlnet, torch_dtype=torch.bfloat16
)
pipe.to("cuda")
# canny
control_image = load_image("https://huggingface.co/InstantX/Qwen-Image-ControlNet-Union/resolve/main/conds/canny.png")
prompt = "Aesthetics art, traditional asian pagoda, elaborate golden accents, sky blue and white color palette, swirling cloud pattern, digital illustration, east asian architecture, ornamental rooftop, intricate detailing on building, cultural representation."
controlnet_conditioning_scale = 1.0
# Please note that the results will not be identical, because the generator is called in different order.
image = pipe(
 prompt=prompt,
 negative_prompt=" ",
 control_image=[control_image, control_image],
 controlnet_conditioning_scale=[controlnet_conditioning_scale/2, controlnet_conditioning_scale/2],
 width=control_image.size[0],
 height=control_image.size[1],
 num_inference_steps=30,
 true_cfg_scale=4.0,
 generator=torch.Generator(device="cuda").manual_seed(42),
).images[0]
image.save(f"qwenimage_cn_union_multi_result_.png")

Sanity Check

J4BEZ, yiyixuxu, vuongminh1907, inFaaa, and HenryPotter0546 reacted with thumbs up emoji J4BEZ reacted with heart emoji tolgacangoz reacted with rocket emoji J4BEZ and yiyixuxu reacted with eyes emoji

# 3. Prepare control image
num_channels_latents = self.transformer.config.in_channels // 4
if isinstance(self.controlnet, QwenImageControlNetModel):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ohh, do we not support multi-controlnets use case?

Comment on lines +328 to +332
if len(self.nets) == 1:
controlnet = self.nets[0]

for i, (image, scale) in enumerate(zip(controlnet_cond, conditioning_scale)):
block_samples = controlnet(
Copy link
Collaborator

@yiyixuxu yiyixuxu Aug 22, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can probably do something like this, and this way we don't need to something special for one union controlnet

Suggested change
if len(self.nets) == 1:
controlnet = self.nets[0]
for i, (image, scale) in enumerate(zip(controlnet_cond, conditioning_scale)):
block_samples = controlnet(
if len(self.nets) == 1:
controlnets = [self.nets] *len(controlnet_cond)
for i, (image, scale, controlnet) in enumerate(zip(controlnet_cond, conditioning_scale, controlnets)):
block_samples = controlnet(

Copy link
Contributor Author

@yiyixuxu Added support for multiple conditions. ruff check has passed.
yiyixuxu reacted with thumbs up emoji

Copy link
Contributor

Excellent work @haofanwang ! did you try controlnet with Flux Kontext too?

Copy link
Collaborator

@bot /style

Copy link
Contributor

github-actions bot commented Aug 22, 2025 •
edited
Loading

Style bot fixed some files and pushed the changes.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Copy link
Collaborator

@yiyixuxu yiyixuxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

@yiyixuxu yiyixuxu merged commit 561ab54 into huggingface:main Aug 22, 2025
14 checks passed
Copy link
Contributor

Copy link
Contributor Author

Will be released soon.

@DN6 DN6 added the roadmap Add to current release roadmap label Aug 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yiyixuxu yiyixuxu yiyixuxu approved these changes

Assignees
No one assigned
Labels
roadmap Add to current release roadmap
Projects
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

AltStyle によって変換されたページ (->オリジナル) /