-
Notifications
You must be signed in to change notification settings - Fork 422
Support for Flux Controls + Flex.2 #692
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
d8eadb7 to
22ab6f0
Compare
22ab6f0 to
bb30633
Compare
(To change the strength of the control, you can try playing around with the --img-cfg-scale, but this makes inference 3x slower, and without apg, it might produce poor results with the flux control models)
Might be specific to Flex.2-preview unfortunately:
Should be able to control the strength directly (for Flex.2-preview) by scaling the control latents as it is done in the custom pipeline: https://huggingface.co/ostris/Flex.2-preview/blob/main/pipeline.py#L262-L263
Might be specific to Flex.2-preview unfortunately: Should be able to control the strength directly (for Flex.2-preview) by scaling the control latents as it is done in the custom pipeline: https://huggingface.co/ostris/Flex.2-preview/blob/main/pipeline.py#L262-L263
The same trick seems to work well with flux canny too, even though in the diffusers implementation, there's no way to set control strength. Not sure how Flux depth will handle scaled latents though, I haven't tested it yet, but I don't think it will work.
Edit: it seems to work too.
iwr-redmond
commented
May 28, 2025
Will this PR make it easier to support SDXL control LoRAs in the future?
Will this PR make it easier to support SDXL control LoRAs in the future?
Yes, some of the changes from this PR will help support the control LoRAs (both for SDXL and flux), but there's still quite a lot of work on the LoRA implementation side for these to function (need support for Lora bias, allowing mismatched dimensions between Lora and model weights... Plus the Loras are loaded late into the pipeline, after the control inputs are prepared)
13a8570 to
47665bd
Compare
47665bd to
1f3f763
Compare
1f3f763 to
88a0634
Compare
88a0634 to
cc15e39
Compare
Uh oh!
There was an error while loading. Please reload this page.
Models
Flux Controls:
https://huggingface.co/black-forest-labs/FLUX.1-Canny-dev
https://huggingface.co/black-forest-labs/FLUX.1-Depth-dev
Flux models with built-in controlnet-like conditionning suport (depth or canny depending on the model)
Flex.2 (preview)
https://huggingface.co/ostris/Flex.2-preview
Pruned and de-distilled opensource Flux model with support for inpainting and controlnet-like conditionning (line, pose and depth all supported by a single model).
Usage
To enable control image conditionning, use the
--control-imageargument (same as when using a control net). You can also use--control-strengthto change how much effect the control image should have.