TheNetherWatcher/Vid2Vid-using-Text-prompt

1### Example2 3- region prompt(txt2img / no controlnet)4- region 0 ... 1girl, upper body etc5- region 1 ... ((car)), street, road,no human etc6- background ... town, outdoors etc7- ip adapter input for background / region 0 / region 18<img src="https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/118420657/ca355f4b-f4c0-4405-88f4-1c80632e32a6" width="512">9 10- animatediff generate -c config/prompts/region_txt2img.json -W 512 -H 768 -L 32 -C 1611- region 0 mask / region 1 mask / txt2img12 13<div><video controls src="https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/118420657/88eb1572-2772-4d76-89c1-c6bf8142283d" muted="false"></video></div>14 15 16 17<div><video controls src="https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/118420657/55042dff-4e23-43b9-b4d6-6f2228b943d2" muted="false"></video></div>18<br>19 20- apply different lora for each region.21- [abdiel](https://civitai.com/models/159943/abdiel-shin-megami-tensei-v-v) for region 022- [amanozoko](https://civitai.com/models/159933/amanozoko-shin-megami-tensei-v-v) for region 123- no lora for background24<div><video controls src="https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/118420657/a5260cf0-1f96-4c65-8408-805428d2528e" muted="false"></video></div>25 26```json27  # new lora_map format28  "lora_map": {29        # Specify lora as a path relative to /animatediff-cli/data30        "share/Lora/zs_Abdiel.safetensors": {   # setting for abdiel lora31            "region" : ["0"],            # target region. Multiple designations possible32            "scale" : {33                # "frame_no" : scale format34                "0": 0.75           # lora scale. same as prompt_map format. For example, it is possible to set the lora to be used from the 30th frame.35            }36        },37        "share/Lora/zs_Amanazoko.safetensors": {  # setting for amanozako lora38            "region" : ["1"],            # target region39            "scale" : {40                "0": 0.7541            }42        }43  },44```45- more example [here](https://github.com/s9roll7/animatediff-cli-prompt-travel/issues/147)46<br>47 48 49 50- img2img51- This can be improved using controlnet, but this sample does not use it.52- source / denoising_strength 0.7 / denoising_strength 0.8553<div><video controls src="https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/118420657/4a9ecf3e-d1c7-468a-a85f-abce7b4c4aab" muted="false"></video></div>54<br>55<br>56 57- [A command to stylization with region has been added](https://github.com/s9roll7/animatediff-cli-prompt-travel#video-stylization-with-region).58- (You can also create json manually without using the stylize command.)59- region prompt60- Region division into person shapes61- source / img2img / txt2img62<div><video controls src="https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/118420657/10101ab8-39cc-4dbc-85c8-af558f80c6fe" muted="false"></video></div>63<br>64 65- source / Region division into person shapes / inpaint66<div><video controls src="https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/118420657/d9231c8e-94b6-4608-97ba-6ab3fc85bcfa" muted="false"></video></div>67<br>68<br>69 70 71 72 73- [A command to stylization with mask has been added](https://github.com/s9roll7/animatediff-cli-prompt-travel#video-stylization-with-mask).74- more example [here](https://github.com/s9roll7/animatediff-cli-prompt-travel/issues/111)75 76<div><video controls src="https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/118420657/461cd68a-511a-4ad8-a2ee-7078faed7354" muted="false"></video></div>77<br>78 79 80- [A command to automate video stylization has been added](https://github.com/s9roll7/animatediff-cli-prompt-travel#video-stylization).81- Original / First generation result / Second generation(for upscaling) result82- It took 4 minutes to generate the first one and about 5 minutes to generate the second one (on rtx 4090).83- more example [here](https://github.com/s9roll7/animatediff-cli-prompt-travel/issues/29)84 85<div><video controls src="https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/118420657/2f1965f2-9a50-485e-ac95-e888a3189ba2" muted="false"></video></div>86<br>87 88 89- controlnet_openpose + controlnet_softedge90- input frames for controlnet(0,16,32 frames)91<img src="https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/118420657/4adac698-75a4-4c6d-bf64-a5723d0e3e77" width="512">92 93- result94<div><video controls src="https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/118420657/50aa9d0d-15b6-4c84-a497-8d020d3bdb7c" muted="false"></video></div>95<br>96 97- In the latest version, generation can now be controlled more precisely through prompts.98- sample 199```json100    "prompt_fixed_ratio": 0.8,101    "head_prompt": "1girl, wizard, circlet, earrings, jewelry, purple hair,",102    "prompt_map": {103        "0": "(standing,full_body),blue_sky, town",104        "8": "(sitting,full_body),rain, town",105        "16": "(standing,full_body),blue_sky, woods",106        "24": "(upper_body), beach",107        "32": "(upper_body, smile)",108        "40": "(upper_body, angry)",109        "48": "(upper_body, smile, from_above)",110        "56": "(upper_body, angry, from_side)",111        "64": "(upper_body, smile, from_below)",112        "72": "(upper_body, angry, from_behind, looking at viewer)",113        "80": "face,looking at viewer",114        "88": "face,looking at viewer, closed_eyes",115        "96": "face,looking at viewer, open eyes, open_mouth",116        "104": "face,looking at viewer, closed_eyes, closed_mouth",117        "112": "face,looking at viewer, open eyes,eyes, open_mouth, tongue, smile, laughing",118        "120": "face,looking at viewer, eating, bowl,chopsticks,holding,food"119    },120```121<div><video controls src="https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/118420657/c4de4b87-f302-4d61-98c7-9607dece386f" muted="false"></video></div>122<br>123 124- sample 2125```json126    "prompt_fixed_ratio": 1.0,127    "head_prompt": "1girl, wizard, circlet, earrings, jewelry, purple hair,",128    "prompt_map": {129        "0": "",130        "8": "((fire magic spell, fire background))",131        "16": "((ice magic spell, ice background))",132        "24": "((thunder magic spell, thunder background))",133        "32": "((skull magic spell, skull background))",134        "40": "((wind magic spell, wind background))",135        "48": "((stone magic spell, stone background))",136        "56": "((holy magic spell, holy background))",137        "64": "((star magic spell, star background))",138        "72": "((plant magic spell, plant background))",139        "80": "((meteor magic spell, meteor background))"140    },141```142<div><video controls src="https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/118420657/31a5827d-e551-4937-8b67-51747a92d14c" muted="false"></video></div>143<br>144 145