models stabilityai stable diffusion 2 inpainting

stabilityai-stable-diffusion-2-inpainting

Overview

stabilityai/stable-diffusion-2-inpainting model is a continuation of the stable-diffusion-2-base model, with an additional 200,000 steps of training. It utilizes a mask-generation strategy introduced in LAMA and combines this with latent Variational Autoencoder (VAE) representations of the masked image to provide additional conditioning during the training process. LAION-5B and subsets are used as training data, which is further filtered using LAION's NSFW detector, with a "p_unsafe" score of 0.1 (conservative). The model should not be used to create harmful, offensive, or discriminatory content. Additionally, the model has limitations, such as difficulties with photorealism, rendering legible text, and generating complex compositions. The model's training data, LAION-5B, primarily contains English descriptions, which can lead to biases and limitations in generating non-English content. To enhance safety, a Safety Checker is recommended for use with this model.

The above summary was generated using ChatGPT. Review the original-model-card to understand the data used to train the model, evaluation metrics, license, intended uses, limitations and bias before using the model.

Inference samples

Inference type	Python sample (Notebook)	CLI with YAML
Real time	text-to-image-inpainting-online-endpoint.ipynb	text-to-image-inpainting-online-endpoint.sh
Batch	text-to-image-inpainting-batch-endpoint.ipynb	text-to-image-inpainting-batch-endpoint.sh

Inference with Azure AI Content Safety (AACS) samples

Inference type	Python sample (Notebook)
Real time	safe-text-to-image-inpainting-online-deployment.ipynb
Batch	safe-text-to-image-inpainting-batch-endpoint.ipynb

Sample inputs and outputs (for real-time inference)

Sample input

{
   "input_data": {
        "columns": ["prompt", "image", "mask"],
        "data": [
            {
                "prompt": "Face of a yellow cat, high resolution, sitting on a park bench",
                "image": "image1",
                "mask_image": "mask1"
            },
            {
                "prompt": "Face of a green cat, high resolution, sitting on a park bench",
                "image": "image2",
                "mask_image": "mask2"
            }
        ],
        "index": [0, 1]
    }
}

Note:

• "image1" and "image2" strings are base64 format.
• "mask1" and "mask2" strings are base64 format.

Sample output

[
    {
        "prompt": "Face of a yellow cat, high resolution, sitting on a park bench",
        "generated_image": "inpainted_image1",
        "nsfw_content_detected": null
    },
    {
        "prompt": "Face of a green cat, high resolution, sitting on a park bench",
        "generated_image": "inpainted_image2",
        "nsfw_content_detected": null
    }
]

Note:

• "inpainted_image1" and "inpainted_image2" strings are base64 format.
• The stabilityai-stable-diffusion-2-inpainting model doesn't check for the NSFW content in generated image. We highly recommend to use the model with Azure AI Content Safety (AACS). Please refer sample online and batch notebooks for AACS integrated deployments.

Model inference: visualization for the prompt - "a small flower vase featuring a blend of yellow and orange"

Version: 4

Tags

Preview license : creativeml-openrail++-m task : text-to-image

View in Studio: https://ml.azure.com/registries/azureml/models/stabilityai-stable-diffusion-2-inpainting/version/4

License: creativeml-openrail++-m

Properties

SHA: 81a84f49b15956b60b4272a405ad3daef3da4590

datasets: LAION-5B

inference-min-sku-spec: 4|1|28|176

inference-recommended-sku: Standard_NC6s_v3, Standard_NC12s_v3, Standard_NC24s_v3, Standard_NC24rs_v3, Standard_NC16as_T4_v3, Standard_NC24ads_A100_v4, Standard_NC48ads_A100_v4, Standard_NC4as_T4_v3, Standard_NC64as_T4_v3, Standard_NC8as_T4_v3, Standard_NC96ads_A100_v4, Standard_ND40rs_v2, Standard_ND96amsr_A100_v4, Standard_ND96asr_v4

model_id: stabilityai/stable-diffusion-2-inpainting