Using Region Control with Hook nodes

[duchamps0305]

Hi, I've been using the new hook loras and prop set combine nodes from the comfy team to do regional conditioning while waiting for for the new update for the ControlAltAI nodes, I was wondering if there is a way to use the great bbox masking system you made using the comfy nodes? My biggest issue right now is creating good enough masks with other nodes while your system seemed to be really promising.

https://blog.comfy.org/p/masking-and-scheduling-lora-and-model-weights

[gseth]

I have tried for the last whole month. The issue is that each Lora key is different, and they don't match the spatial control level in the flux pipeline.

I made two different pipelines and around 20 new nodes, tried all combinations, and kept running into issues.

The basic concept would be to take the mask and convert it to latent space, which is successfully done. The comfy default lora hook doesn't work. So I created a new lora hook set it up set clip hook. We then forward that to region mask conditioning from pipeline. Which was modified to hold the dict values of the Conditioning bbox, mask and lora. We then pass everything to the flux attention control.

Over here the issue happens. I end up with Mis Match issues. I cannot fix it.

For this we need to completely reverse engineer the flux + lora interaction. which will take months and by that time we will have something new, unless the team at black forest labs just advise on how to proceed at this stage.

The simplest way is a Left and right Region, Lora, but then that's not this pipeline, and the results are underwhelming. To get LoRA on this level of spatial control would be great. I don't mind sharing the actual logic in the flux attention, if someone can take a look at it, some one who has a deeper knowledge and solve the issue.

New Nodes plus test workflow (not working: https://drive.google.com/file/d/1TYCH2q_BARjZh4690lGYgOFy7IX5L4xW/view?usp=sharing

At this point I have given up, unless someone can help here.