Disty0/sote-diffusion-cascade_pre-alpha0

1---2pipeline_tag: text-to-image3license: other4license_name: stable-cascade-nc-community5license_link: LICENSE6---7 8# SoteDiffusion Cascade9 10Anime finetune of Stable Cascade.  11Currently is in very early state in training.  12No commercial use thanks to StabilityAI.  13 14<style>15.image {16    float: left;17    margin-left: 10px;18}19</style>20 21<table>22<img class="image" src="https://cdn-uploads.huggingface.co/production/uploads/6456af6195082f722d178522/DVcAEhQr_FarvoLawYpBM.png" width="320">23<img class="image" src="https://cdn-uploads.huggingface.co/production/uploads/6456af6195082f722d178522/kNts3NhZogHHqC5JfKRkr.png" width="320">24</table>25 26## Code Example27 28```shell29pip install diffusers30```31 32```python33import torch34from diffusers import StableCascadeDecoderPipeline, StableCascadePriorPipeline35 36prompt = "(extremely aesthetic, best quality, newest), 1girl, solo, cat ears, looking at viewer, blush, light smile, upper body,"37negative_prompt = "very displeasing, worst quality, monochrome, sketch, blurry, fat, child,"38 39prior = StableCascadePriorPipeline.from_pretrained("Disty0/sote-diffusion-cascade_pre-alpha0", torch_dtype=torch.float16)40decoder = StableCascadeDecoderPipeline.from_pretrained("Disty0/sote-diffusion-cascade-decoder_pre-alpha0", torch_dtype=torch.float16)41 42prior.enable_model_cpu_offload()43prior_output = prior(44    prompt=prompt,45    height=1024,46    width=1024,47    negative_prompt=negative_prompt,48    guidance_scale=6.0,49    num_images_per_prompt=1,50    num_inference_steps=4051)52 53decoder.enable_model_cpu_offload()54decoder_output = decoder(55    image_embeddings=prior_output.image_embeddings,56    prompt=prompt,57    negative_prompt=negative_prompt,58    guidance_scale=2.0,59    output_type="pil",60    num_inference_steps=1061).images[0]62decoder_output.save("cascade.png")63```64 65 66## Training Status:67 68**GPU used for training**: 1x AMD RX 7900 XTX 24GB69 70| dataset name | training done | remaining |71|---|---|---|72| **newest** | 002 | 218 |73| **late** | 002 | 204 |74| **mid** | 002 | 199 |75| **early** | 002 | 053 |76| **oldest** | 002 | 014 |77| **pixiv** | 002 | 072 |78| **visual novel cg** | 002 | 068 |79| **anime wallpaper** | 002 | 011 |80| **Total** | 24 | 839 |81 82**Note**: chunks starts from 0 and there are 8000 images per chunk  83 84 85## Dataset:86 87**GPU used for captioning**: 1x Intel ARC A770 16GB  88**Model used for captioning**: SmilingWolf/wd-v1-4-convnextv2-tagger-v2  89 90 91| dataset name | total images | total chunk |92|---|---|---|93| **newest** | 1.766.335 | 221 |94| **late** | 1.652.420 | 207 |95| **mid** | 1.609.608 | 202 |96| **early** | 442.368 | 056 |97| **oldest** | 128.311 | 017 |98| **pixiv** | 594.046 | 075 |99| **visual novel cg** | 560.903 | 071 |100| **anime wallpaper** | 106.882 | 014 |101| **Total** | 6.860.873 | 863 |102 103**Note**: Smallest size is 1280x600 | 768.000 pixels104 105 106## Tags:107 108```109aesthetic tags, quality tags, date tags, custom tags, rest of the tags110```111 112### Date:113| tag | date |114|---|---|115| **newest** | 2022 to 2024 |116| **late** | 2019 to 2021 |117| **mid** | 2015 to 2018 |118| **early** | 2011 to 2014 |119| **oldest** | 2005 to 2010 |120 121### Aesthetic Tags:122 123**Model used**: shadowlilac/aesthetic-shadow124 125| score greater than | tag |126|---|---|127| **0.980** | extremely aesthetic |128| **0.900** | very aesthetic |129| **0.750** | aesthetic |130| **0.500** | slightly aesthetic |131| **0.350** | not displeasing |132| **0.250** | not aesthetic |133| **0.125** | slightly displeasing |134| **0.025** | displeasing |135| **rest of them** | very displeasing |136 137### Quality Tags:138 139**Model used**: https://huggingface.co/hakurei/waifu-diffusion-v1-4/blob/main/models/aes-B32-v0.pth140 141 142| score greater than | tag |143|---|---|144| **0.980** | best quality |145| **0.900** | high quality |146| **0.750** | great quality |147| **0.500** | medium quality |148| **0.250** | normal quality |149| **0.125** | bad quality |150| **0.025** | low quality |151| **rest of them** | worst quality |152 153## Custom Tags:154 155| dataset name | custom tag |156|---|---|157| **image boards** | date, |158| **pixiv** | art by Display_Name, |159| **visual novel cg** | Full_VN_Name (short_3_letter_name), visual novel cg, |160| **anime wallpaper** | date, anime wallpaper, |161 162## Training Params:163 164**Software used**: Kohya SD-Scripts with Stable Cascade branch  165**Base model**: KBlueLeaf/Stable-Cascade-FP16-fixed  166 167### Command:168```169accelerate launch  --mixed_precision fp16 --num_cpu_threads_per_process 1 stable_cascade_train_stage_c.py \170--mixed_precision fp16 \171--save_precision fp16 \172--full_fp16 \173--sdpa \174--gradient_checkpointing \175--resolution "1024,1024" \176--train_batch_size 2 \177--gradient_accumulation_steps 32 \178--adaptive_loss_weight \179--learning_rate 4e-6 \180--lr_scheduler constant_with_warmup \181--lr_warmup_steps 100 \182--optimizer_type adafactor \183--optimizer_args "scale_parameter=False" "relative_step=False" "warmup_init=False" \184--max_grad_norm 0 \185--token_warmup_min 1 \186--token_warmup_step 0 \187--shuffle_caption \188--caption_dropout_rate 0 \189--caption_tag_dropout_rate 0 \190--caption_dropout_every_n_epochs 0 \191--dataset_repeats 1 \192--save_state \193--save_every_n_steps 128 \194--sample_every_n_steps 32 \195--max_token_length 225 \196--max_train_epochs 1 \197--caption_extension ".txt" \198--max_data_loader_n_workers 2 \199--persistent_data_loader_workers \200--enable_bucket \201--min_bucket_reso 256 \202--max_bucket_reso 4096 \203--bucket_reso_steps 64 \204--bucket_no_upscale \205--log_with tensorboard \206--output_name sotediffusion-sc_3b \207--train_data_dir /mnt/DataSSD/AI/anime_image_dataset/combined/combined-0002 \208--in_json /mnt/DataSSD/AI/anime_image_dataset/combined/combined-0002.json \209--output_dir /mnt/DataSSD/AI/SoteDiffusion/StableCascade/sotediffusion-sc_3b-2 \210--logging_dir /mnt/DataSSD/AI/SoteDiffusion/StableCascade/sotediffusion-sc_3b-2/logs \211--resume /mnt/DataSSD/AI/SoteDiffusion/StableCascade/sotediffusion-sc_3b-1/sotediffusion-sc_3b-1-state \212--stage_c_checkpoint_path /mnt/DataSSD/AI/SoteDiffusion/StableCascade/sotediffusion-sc_3b-1/sotediffusion-sc_3b-1.safetensors \213--effnet_checkpoint_path /mnt/DataSSD/AI/models/sd-cascade/effnet_encoder.safetensors \214--previewer_checkpoint_path /mnt/DataSSD/AI/models/sd-cascade/previewer.safetensors \215--sample_prompts /mnt/DataSSD/AI/SoteDiffusion/StableCascade/sotediffusion-prompt.txt216```217 218 219## Limitations and Bias220 221### Bias222 223- This model is intended for anime illustrations.  224  Realistic capabilites are not tested at all.  225- Current version has bias to older anime styles.  226 227### Limitations228- Can fall back to realistic.  229  Use "anime illustration" tag to point it into the right direction.  230- Far shot eyes are bad thanks to the heavy latent compression.  231