Skip to content

Commit

Permalink
Merge branch 'master' of https://github.com/ml-research/diffusion-web…
Browse files Browse the repository at this point in the history 
…page

# Conflicts:
#	.idea/misc.xml
#	.idea/vcs.xml
  • Loading branch information
@felifri
felifri committed Jun 29, 2023
2 parents 64a45cc + e13313e commit eab0c46
Show file tree
Hide file tree
Showing 20 changed files with 5,233 additions and 644 deletions.
2 changes: 1 addition & 1 deletion .idea/misc.xml
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion .idea/vcs.xml
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 12 additions & 0 deletions .nojekyll
View file
Original file line number Diff line number Diff line change
@@ -1 +1,13 @@

@misc{brack2022Stable,
Anote = {./images/sega_graphic.png},
title={The Stable Artist: Steering Semantics in Diffusion Latent Space},
author={Manuel Brack and Patrick Schramowski and Felix Friedrich and Dominik Hintersdorf and Kristian Kersting},
Howpublished = {arXiv preprint arXiv:2212.06013},
year = {2022},
month={Dez},
Note = {Large, text-conditioned generative diffusion models have recently gained a lot of attention for their impressive performance in generating high-fidelity images from text alone. However, achieving high-quality results is almost unfeasible in a one-shot fashion. On the contrary, text-guided image generation involves the user making many slight changes to inputs in order to iteratively carve out the envisioned image. However, slight changes to the input prompt often lead to entirely different images being generated, and thus the control of the artist is limited in its granularity. To provide flexibility, we present the Stable Artist, an image editing approach enabling fine-grained control of the image generation process. The main component is semantic guidance (SEGA) which steers the diffusion process along variable numbers of semantic directions. This allows for subtle edits to images, changes in composition and style, as well as optimization of the overall artistic conception. Furthermore, SEGA enables probing of latent spaces to gain insights into the representation of concepts learned by the model, even complex ones such as 'carbon emission'. We demonstrate the Stable Artist on several tasks, showcasing high-quality image editing and composition.},
Pages = {},
Keywords = {Representations, Text-to-Image Synthesis, Text-Guided Image Generation, Stable Diffusion, Concepts, Semantics},
Url={https://arxiv.org/abs/2212.06013}
}
922 changes: 644 additions & 278 deletions index.html
View file

Large diffs are not rendered by default.

280 changes: 280 additions & 0 deletions projects/backdoor/index.html
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,280 @@
<!DOCTYPE html>
<html>

<head>
<meta charset="utf-8">
<!-- Meta tags for social media banners, these should be filled in appropriatly as they are your "business card" -->
<!-- Replace the content tag with appropriate information -->
<meta name="description" content="Rickrolling the Artist - Project Page">
<meta property="og:title" content="Rickrolling the Artist - Project Page" />
<meta property="og:description" content="" />
<meta property="og:url" content="URL OF THE WEBSITE" />
<!-- Path to banner image, should be in the path listed below. Optimal dimenssions are 1200X630-->
<meta property="og:image" content="/diffusion-webpage/static/image/your_banner_image.png" />
<meta property="og:image:width" content="1200" />
<meta property="og:image:height" content="630" />


<meta name="twitter:title" content="Rickrolling the Artist - Project Page">
<meta name="twitter:description" content="">
<!-- Path to banner image, should be in the path listed below. Optimal dimenssions are 1200X600-->
<meta name="twitter:image" content="/diffusion-webpage/static/images/your_twitter_banner_image.png">
<meta name="twitter:card" content="summary_large_image">
<!-- Keywords for your paper to be indexed by-->
<meta name="keywords"
content="safe latent diffusion, image generation, text-to-image, stable diffusion, imagen, dall-e">
<meta name="viewport" content="width=device-width, initial-scale=1">


<title>Rickrolling the Artist</title>
<link rel="icon" type="image/x-icon" href="/diffusion-webpage/static/images/favicon.png">
<link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" rel="stylesheet">

<link rel="stylesheet" href="/diffusion-webpage/static/css/bulma.min.css">
<link rel="stylesheet" href="/diffusion-webpage/static/css/bulma-carousel.min.css">
<link rel="stylesheet" href="/diffusion-webpage/static/css/bulma-slider.min.css">
<link rel="stylesheet" href="/diffusion-webpage/static/css/fontawesome.all.min.css">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
<link rel="stylesheet" href="/diffusion-webpage/static/css/index.css">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/10.0.3/styles/default.min.css">
<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/10.0.3/highlight.min.js"></script>


<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
<script src="https://documentcloud.adobe.com/view-sdk/main.js"></script>
<script defer src="/diffusion-webpage/static/js/fontawesome.all.min.js"></script>
<script src="/diffusion-webpage/static/js/bulma-carousel.min.js"></script>
<script src="/diffusion-webpage/static/js/bulma-slider.min.js"></script>
<script src="/diffusion-webpage/static/js/index.js"></script>
<script>hljs.initHighlightingOnLoad();</script>
</head>

<body>
<div style="margin: 10px">
<a href="/diffusion-webpage/index.html" class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fa fa-home"></i>
</span>
<span>Home</span>
</a>
</div>
<section class="hero">
<div class="hero-body">
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column has-text-centered">
<h1 class="title is-1 publication-title">Rickrolling the Artist:<br>Injecting Backdoors into
Text Encoders for Text-to-Image Synthesis</h1>
<div class="is-size-5 publication-authors">
<!-- Paper authors -->
<span class="author-block">
<a href="https://www.aiml.informatik.tu-darmstadt.de/people/lstruppek"
target="_blank">Lukas Struppek</a>,</span>
<span class="author-block">
<a href="https://www.aiml.informatik.tu-darmstadt.de/people/dhintersdorf"
target="_blank">Dominik Hintersdorf</a>,</span>

<span class="author-block">
<a href="https://www.aiml.informatik.tu-darmstadt.de/people/kkersting"
target="_blank">Kristian Kersting</a>
</span>
</div>

<div class="is-size-5 publication-authors">
<span class="author-block">TU Darmstadt<br></span>
</div>

<div class="column has-text-centered">
<div class="publication-links">
<!-- Arxiv PDF link -->
<span class="link-block">
<a href="https://arxiv.org/pdf/2211.02408.pdf" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fas fa-file-pdf"></i>
</span>
<span>Paper</span>
</a>
</span>



<!-- Github link -->
<span class="link-block">
<a href="https://github.com/lukasstruppek/rickrolling-the-artist" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fab fa-github"></i>
</span>
<span>Code</span>
</a>
</span>

<!-- ArXiv abstract Link -->
<span class="link-block">
<a href="https://arxiv.org/abs/2211.02408" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="ai ai-arxiv"></i>
</span>
<span>arXiv</span>
</a>
</span>

</div>
</div>
</div>
</div>
</div>
</div>
</section>


<!-- Paper abstract -->
<section class="section hero is-light">
<div class="container is-max-desktop">
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">Abstract</h2>
<div class="content has-text-justified">
<p>
While text-to-image synthesis currently enjoys great popularity among researchers and the
general public, the security of these models has been neglected so far. Many text-guided
image generation models rely on pre-trained text encoders from external sources, and their
users trust that the retrieved models will behave as promised. Unfortunately, this might not
be the case. We introduce backdoor attacks against text-guided generative models and
demonstrate that their text encoders pose a major tampering risk. Our attacks only slightly
alter an encoder so that no suspicious model behavior is apparent for image generations with
clean prompts. By then inserting a single character trigger into the prompt, e.g., a
non-Latin character or emoji, the adversary can trigger the model to either generate images
with pre-defined attributes or images following a hidden, potentially malicious description.
We empirically demonstrate the high effectiveness of our attacks on Stable Diffusion and
highlight that the injection process of a single backdoor takes less than two minutes.
Besides phrasing our approach solely as an attack, it can also force an encoder to forget
phrases related to certain concepts, such as nudity or violence, and help to make image
generation safer. </p>
</div>
</div>
</div>
</div>
</section>
<!-- End paper abstract -->


<section class="hero is-small">
<div class="hero-body">
<div class="container">
<h2 class="title is-3">Backdoor Concept</h2>
<div id="">
<div class="item">
<!-- Your image here -->
<img src="/diffusion-webpage/static/images/rickrolling_concept.jpg" />
<h3 class="subtitle has-text-centered">
Concept of our backdoor attack against CLIP-based text-to-image synthesis models, in this
case, Stable Diffusion. We
fine-tune the CLIP text encoder to integrate the backdoors and leave all other model
components untouched. The poisoned
text encoder is then spread over the internet, e.g., by domain name spoofing attacks - pay
attention to the model URL!
In the depicted case, inserting a single inconspicuous trigger character, a Cyrillic o,
forces the model to generate images of Rick Astley instead of boats on a
lake.
</h3>

</div>
</div>
</div>
</div>
</section>

<section class="hero is-small">
<div class="hero-body">
<div class="container">
<h2 class="title is-3">Examples with Homoglyphs as Triggers</h2>
<div id="">
<div class="item">
<!-- Your image here -->
<img src="/diffusion-webpage/static/images/backdoor_samples.jpg" />
<h3 class="subtitle has-text-centered">
Generated samples of the clean and poisoned models. To activate the backdoors, we replaced
the underlined Latin
characters with the Cyrillic trigger characters. Each column corresponds to a different
prompt. The bottom row shows results for the poisoned encoder with triggers in
the prompts
</h3>

</div>
</div>
</div>
</div>
</section>

<section class="hero is-small">
<div class="hero-body">
<div class="container">
<h2 class="title is-3">Backdoors Can Also Erase Concepts</h2>
<div id="">
<div class="item">
<!-- Your image here -->
<img src="/diffusion-webpage/static/images/safe_samples_backdoor.jpg" />
<h3 class="subtitle has-text-centered">
Remapping concepts associated with nudity to an empty string. It avoids explicit content
generation triggered by
specific words. We fine-tuned the poisoned encoder to map the underlined words to an empty
string.
</h3>

</div>
</div>
</div>
</div>
</section>

<!--BibTex citation -->
<section class="section" id="BibTeX">
<div class="container is-max-desktop content">
<h2 class="title">BibTeX</h2>
<pre><code>@article{struppek2022rickrolling,
title={Rickrolling the Artist: Injecting Backdoors into Text-Guided Image Generation Models},
author={Lukas Struppek and Dominik Hintersdorf and Kristian Kersting},
year={2022},
journal={arXiv preprint at arXiv:2211.02408}
}</code></pre>
</div>
</section>
<!--End BibTex citation -->


<footer class="footer">
<div class="container">
<div class="columns is-centered">
<div class="column is-8">
<div class="content">

<p>
This page was built using the <a
href="https://github.com/eliahuhorwitz/Academic-project-page-template"
target="_blank">Academic
Project Page Template</a>.
You are free to borrow the of this website, we just ask that you link back to this page in
the
footer. <br> This website is licensed under a <a rel="license"
href="http://creativecommons.org/licenses/by-sa/4.0/" target="_blank">Creative
Commons Attribution-ShareAlike 4.0 International License</a>.
</p>

</div>
</div>
</div>
</div>
</footer>

<!-- Statcounter tracking code -->

<!-- You can add a tracker to track page visits by creating an account at statcounter.com -->

<!-- End of Statcounter Code -->

</body>

</html>
Loading

0 comments on commit eab0c46

Please sign in to comment.