Toward effective protection against diffusion-based mimicry through score distillation [ICLR'2024]

Haotian Xue ¹, Chumeng Liang ^2,3, Xiaoyu Wu ², and Yongxin Chen ¹

¹ Georgia Tech ² SJTU ³ USC

TL;DR : novel insights into attacks agasint LDM, a more effective protection against malicious diffusion model editing

Updates

• [01/15/2024] 🎉 Our paper is accepted to ICLR'2024!
• [11/24/2023] Paper finally shown on Arxiv
• [09/27/2023] Paper will be released soon!
• [09/27/2023] Our repo is alive!

Quick Setup

Install the envs:

conda env create -f env.yml
conda activate mist
pip install --force-reinstall pillow

Download the checkpoint of LDM [Stable-diffusion-model v1.4 checkpoint]

wget -c https://huggingface.co/CompVis/stable-diffusion-v-1-4-original/resolve/main/sd-v1-4.ckpt
mkdir ckpt
mv sd-v1-4.ckpt ckpt/model.ckpt

Run the Protection

Configs can be set in configs/attack/base.yaml, some keys arguments include:

attack:
    epsilon: 16 # l_inf budget
    steps: 100 # attack steps
    input_size: 512 # image size
    mode: sds 
    img_path: [IMAGE_DIR_PATH]
    output_path: [OUT_PATH]
    alpha: 1 # step size
    g_mode: "+"
    use_wandb: False
    project_name: iclr_mist
    diff_pgd: [False, 0.2, 'ddim100'] # un-used feature
    using_target: False
    target_rate: 5 # scale factor for L_S and L_T
    device: 0 # GPU id

here we offer some scripts to run different type of protections:

AdvDM:

python code/diff_mist.py attack.mode='advdm' attack.g_mode='+'

PhotoGuard:

python code/diff_mist.py attack.mode='texture_only' attack.g_mode='+'

Mist:

python code/diff_mist.py attack.mode='mist' attack.g_mode='+'

AdvDM(-):

python code/diff_mist.py attack.mode='advdm' attack.g_mode='-'

SDS(+):

python code/diff_mist.py attack.mode='sds' attack.g_mode='+'

SDS(-):

python code/diff_mist.py attack.mode='sds' attack.g_mode='-'

SDST(-):

python code/diff_mist.py attack.mode='sds' attack.g_mode='-' attack.using_target=True

the output includes: [NAME]_attacked.png which is the attacked image, [NAME]_multistep.png which is the SDEdit results, and [NAME]_onestep.png which is the onestep x_0 prediction results.

[From let to right]: AdvDM, Mist, SDS(-), using eps=16, SDS-version is much more effective than the previous two methods

Cited as:

@inproceedings{xue2023toward,
  title={Toward effective protection against diffusion-based mimicry through score distillation},
  author={Xue, Haotian and Liang, Chumeng and Wu, Xiaoyu and Chen, Yongxin},
  booktitle={The Twelfth International Conference on Learning Representations},
  year={2023}
}