example.md

Example

• region prompt(txt2img / no controlnet)
• region 0 ... 1girl, upper body etc
• region 1 ... ((car)), street, road,no human etc
• background ... town, outdoors etc
• ip adapter input for background / region 0 / region 1

• animatediff generate -c config/prompts/region_txt2img.json -W 512 -H 768 -L 32 -C 16
• region 0 mask / region 1 mask / txt2img

region_txt2img_sample.mp4

region_txt2img_sample2.mp4

• apply different lora for each region.
• abdiel for region 0
• amanozoko for region 1
• no lora for background

region_lora_txt2img_sample.mp4

  # new lora_map format
  "lora_map": {
        # Specify lora as a path relative to /animatediff-cli/data
        "share/Lora/zs_Abdiel.safetensors": {   # setting for abdiel lora
            "region" : ["0"],            # target region. Multiple designations possible
            "scale" : {
                # "frame_no" : scale format
                "0": 0.75           # lora scale. same as prompt_map format. For example, it is possible to set the lora to be used from the 30th frame.
            }
        },
        "share/Lora/zs_Amanazoko.safetensors": {  # setting for amanozako lora
            "region" : ["1"],            # target region
            "scale" : {
                "0": 0.75
            }
        }
  },

• more example here

• img2img
• This can be improved using controlnet, but this sample does not use it.
• source / denoising_strength 0.7 / denoising_strength 0.85

img2img_sample.mp4

• A command to stylization with region has been added.
• (You can also create json manually without using the stylize command.)
• region prompt
• Region division into person shapes
• source / img2img / txt2img

region_sample3.mp4

• source / Region division into person shapes / inpaint

region_sample2.mp4

• A command to stylization with mask has been added.
• more example here

mask_sample1.mp4

• A command to automate video stylization has been added.
• Original / First generation result / Second generation(for upscaling) result
• It took 4 minutes to generate the first one and about 5 minutes to generate the second one (on rtx 4090).
• more example here

style_sample.mp4

• controlnet_openpose + controlnet_softedge
• input frames for controlnet(0,16,32 frames)

• result

output.mp4

• In the latest version, generation can now be controlled more precisely through prompts.
• sample 1

    "prompt_fixed_ratio": 0.8,
    "head_prompt": "1girl, wizard, circlet, earrings, jewelry, purple hair,",
    "prompt_map": {
        "0": "(standing,full_body),blue_sky, town",
        "8": "(sitting,full_body),rain, town",
        "16": "(standing,full_body),blue_sky, woods",
        "24": "(upper_body), beach",
        "32": "(upper_body, smile)",
        "40": "(upper_body, angry)",
        "48": "(upper_body, smile, from_above)",
        "56": "(upper_body, angry, from_side)",
        "64": "(upper_body, smile, from_below)",
        "72": "(upper_body, angry, from_behind, looking at viewer)",
        "80": "face,looking at viewer",
        "88": "face,looking at viewer, closed_eyes",
        "96": "face,looking at viewer, open eyes, open_mouth",
        "104": "face,looking at viewer, closed_eyes, closed_mouth",
        "112": "face,looking at viewer, open eyes,eyes, open_mouth, tongue, smile, laughing",
        "120": "face,looking at viewer, eating, bowl,chopsticks,holding,food"
    },

sample2.mp4

• sample 2

    "prompt_fixed_ratio": 1.0,
    "head_prompt": "1girl, wizard, circlet, earrings, jewelry, purple hair,",
    "prompt_map": {
        "0": "",
        "8": "((fire magic spell, fire background))",
        "16": "((ice magic spell, ice background))",
        "24": "((thunder magic spell, thunder background))",
        "32": "((skull magic spell, skull background))",
        "40": "((wind magic spell, wind background))",
        "48": "((stone magic spell, stone background))",
        "56": "((holy magic spell, holy background))",
        "64": "((star magic spell, star background))",
        "72": "((plant magic spell, plant background))",
        "80": "((meteor magic spell, meteor background))"
    },

sample1.mp4