Using LORA or PEFT on unconditional image generation models

[ChullMotors]

Can anyone help me out with the task of using LORA (/PEFT) on an unconditional image generation model, such as "google/ddpm-celebahq-256"?

This tutorial:
https://huggingface.co/learn/diffusion-course/en/unit2/2
is good for getting going on fine tuning unconditional image generation models, but now I want to add in LORA. Am struggling to figure out or find examples anywhere despite hunting for some time.

The easy route of using the PeftConfig class doesn't seem like it will work as the TaskType class does not support unconditional image generation(as indicated by the TaskType class in peft_types.py), so I can't just pass the model as instantiated with DDIMPipeline through PeftConfig.

Thanks!