Fix: Exclude module handling in LoRA weight loading

[yeonjoon-jung01]

There is a bug in src/diffusers/utils/peft_utils.py when loading LoRA weights: if an excluded module name is a substring of a target module name, it unintentionally prevents the target module from being loaded.

A simple reproduction example:

import torch
from diffusers import FluxPipeline

pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16)
pipe.load_lora_weights("Shakker-Labs/FLUX.1-dev-LoRA-AntiBlur", weight_name="FLUX-dev-lora-AntiBlur.safetensors")

This is a known issue, as discussed in:

• load_lora_weights bug in FLUX.1 diffusers-0.35.0.dev0 #11885
• [lora] exclude_modules won't consider the modules names in target_modules #11874

Although the PEFT library is preparing a fix (see huggingface/peft#2637 (comment)), it's uncertain when that change will be released as part of a stable version. In the meantime, this bug exists in the current main branch of Diffusers and needs to be addressed immediately.

The proposed fix modifies the logic in Diffusers to ensure that exclude_modules are not substring of any target module name, preventing incorrect exclusions. This change may be rolled back in the future once the PEFT-side fix is available and adopted, but until then, it ensures correct and stable behavior for users.

[sayakpaul]

@yeonjoon-jung01 thank you for your contribution but I am afraid I will have to ask you to close the PR as I will open a PR shortly fixing the issue on a deeper level. I hope you will understand and I am sorry about the situation.

[sayakpaul]

#11999

[sayakpaul]

Closing the PR in favor of #11999. Thank you for your contributions, though.