CoolFace
Modelpublic

Disty0/sote-diffusion-cascade_pre-alpha0

sourceHugging Faceotherupdated 2y agoView on Hugging Face
2likes78downloads
Model Card

SoteDiffusion Cascade

Anime finetune of Stable Cascade. Currently is in very early state in training. No commercial use thanks to StabilityAI.

<style> .image { float: left; margin-left: 10px; } </style>

<table> <img class="image" src="https://cdn-uploads.huggingface.co/production/uploads/6456af6195082f722d178522/DVcAEhQr_FarvoLawYpBM.png" width="320"> <img class="image" src="https://cdn-uploads.huggingface.co/production/uploads/6456af6195082f722d178522/kNts3NhZogHHqC5JfKRkr.png" width="320"> </table>

Code Example

shell
pip install diffusers
python
import torch
from diffusers import StableCascadeDecoderPipeline, StableCascadePriorPipeline

prompt = "(extremely aesthetic, best quality, newest), 1girl, solo, cat ears, looking at viewer, blush, light smile, upper body,"
negative_prompt = "very displeasing, worst quality, monochrome, sketch, blurry, fat, child,"

prior = StableCascadePriorPipeline.from_pretrained("Disty0/sote-diffusion-cascade_pre-alpha0", torch_dtype=torch.float16)
decoder = StableCascadeDecoderPipeline.from_pretrained("Disty0/sote-diffusion-cascade-decoder_pre-alpha0", torch_dtype=torch.float16)

prior.enable_model_cpu_offload()
prior_output = prior(
    prompt=prompt,
    height=1024,
    width=1024,
    negative_prompt=negative_prompt,
    guidance_scale=6.0,
    num_images_per_prompt=1,
    num_inference_steps=40
)

decoder.enable_model_cpu_offload()
decoder_output = decoder(
    image_embeddings=prior_output.image_embeddings,
    prompt=prompt,
    negative_prompt=negative_prompt,
    guidance_scale=2.0,
    output_type="pil",
    num_inference_steps=10
).images[0]
decoder_output.save("cascade.png")

Training Status:

GPU used for training: 1x AMD RX 7900 XTX 24GB

dataset nametraining doneremaining
newest002218
late002204
mid002199
early002053
oldest002014
pixiv002072
visual novel cg002068
anime wallpaper002011
Total24839

Note: chunks starts from 0 and there are 8000 images per chunk

Dataset:

GPU used for captioning: 1x Intel ARC A770 16GB Model used for captioning: SmilingWolf/wd-v1-4-convnextv2-tagger-v2

dataset nametotal imagestotal chunk
newest1.766.335221
late1.652.420207
mid1.609.608202
early442.368056
oldest128.311017
pixiv594.046075
visual novel cg560.903071
anime wallpaper106.882014
Total6.860.873863

Note: Smallest size is 1280x600 | 768.000 pixels

Tags:

aesthetic tags, quality tags, date tags, custom tags, rest of the tags

Date:

tagdate
newest2022 to 2024
late2019 to 2021
mid2015 to 2018
early2011 to 2014
oldest2005 to 2010

Aesthetic Tags:

Model used: shadowlilac/aesthetic-shadow

score greater thantag
0.980extremely aesthetic
0.900very aesthetic
0.750aesthetic
0.500slightly aesthetic
0.350not displeasing
0.250not aesthetic
0.125slightly displeasing
0.025displeasing
rest of themvery displeasing

Quality Tags:

Model used: https://huggingface.co/hakurei/waifu-diffusion-v1-4/blob/main/models/aes-B32-v0.pth

score greater thantag
0.980best quality
0.900high quality
0.750great quality
0.500medium quality
0.250normal quality
0.125bad quality
0.025low quality
rest of themworst quality

Custom Tags:

dataset namecustom tag
image boardsdate,
pixivart by Display_Name,
visual novel cgFullVNName (short3letter_name), visual novel cg,
anime wallpaperdate, anime wallpaper,

Training Params:

Software used: Kohya SD-Scripts with Stable Cascade branch Base model: KBlueLeaf/Stable-Cascade-FP16-fixed

Command:

accelerate launch  --mixed_precision fp16 --num_cpu_threads_per_process 1 stable_cascade_train_stage_c.py \
--mixed_precision fp16 \
--save_precision fp16 \
--full_fp16 \
--sdpa \
--gradient_checkpointing \
--resolution "1024,1024" \
--train_batch_size 2 \
--gradient_accumulation_steps 32 \
--adaptive_loss_weight \
--learning_rate 4e-6 \
--lr_scheduler constant_with_warmup \
--lr_warmup_steps 100 \
--optimizer_type adafactor \
--optimizer_args "scale_parameter=False" "relative_step=False" "warmup_init=False" \
--max_grad_norm 0 \
--token_warmup_min 1 \
--token_warmup_step 0 \
--shuffle_caption \
--caption_dropout_rate 0 \
--caption_tag_dropout_rate 0 \
--caption_dropout_every_n_epochs 0 \
--dataset_repeats 1 \
--save_state \
--save_every_n_steps 128 \
--sample_every_n_steps 32 \
--max_token_length 225 \
--max_train_epochs 1 \
--caption_extension ".txt" \
--max_data_loader_n_workers 2 \
--persistent_data_loader_workers \
--enable_bucket \
--min_bucket_reso 256 \
--max_bucket_reso 4096 \
--bucket_reso_steps 64 \
--bucket_no_upscale \
--log_with tensorboard \
--output_name sotediffusion-sc_3b \
--train_data_dir /mnt/DataSSD/AI/anime_image_dataset/combined/combined-0002 \
--in_json /mnt/DataSSD/AI/anime_image_dataset/combined/combined-0002.json \
--output_dir /mnt/DataSSD/AI/SoteDiffusion/StableCascade/sotediffusion-sc_3b-2 \
--logging_dir /mnt/DataSSD/AI/SoteDiffusion/StableCascade/sotediffusion-sc_3b-2/logs \
--resume /mnt/DataSSD/AI/SoteDiffusion/StableCascade/sotediffusion-sc_3b-1/sotediffusion-sc_3b-1-state \
--stage_c_checkpoint_path /mnt/DataSSD/AI/SoteDiffusion/StableCascade/sotediffusion-sc_3b-1/sotediffusion-sc_3b-1.safetensors \
--effnet_checkpoint_path /mnt/DataSSD/AI/models/sd-cascade/effnet_encoder.safetensors \
--previewer_checkpoint_path /mnt/DataSSD/AI/models/sd-cascade/previewer.safetensors \
--sample_prompts /mnt/DataSSD/AI/SoteDiffusion/StableCascade/sotediffusion-prompt.txt

Limitations and Bias

Bias

  • —This model is intended for anime illustrations. Realistic capabilites are not tested at all.
  • —Current version has bias to older anime styles.

Limitations

  • —Can fall back to realistic. Use "anime illustration" tag to point it into the right direction.
  • —Far shot eyes are bad thanks to the heavy latent compression.