ghoskno/Color-Canny-Controlnet-model

Color-Canny CantrolNet

These are ControlNet checkpoints trained on runwayml/stable-diffusion-v1-5, using fused color and canny edge as conditioning.

You can find some example images in the following.

Examples

Color examples

prompt: a concept art of by Makoto Shinkai, a girl is standing in the middle of the sea

negative prompt: text, bad anatomy, blurry, (low quality, blurry) [image]

prompt: a concept art of by Makoto Shinkai, a girl is standing in the middle of the sea

negative prompt: text, bad anatomy, blurry, (low quality, blurry) [image]

prompt: a concept art of by Makoto Shinkai, a girl is standing in the middle of the grass

negative prompt: text, bad anatomy, blurry, (low quality, blurry) [image]

Brightness examples

This model also can be used to control image brightness. The following images are generated with different brightness conditioning image and controlnet strength(0.5 ~ 0.7). [image]

Limitations and Bias

• —No strict control by input color
• —Sometimes generate image with confusion When color description in prompt

Training

Dataset We train this model on laion-art dataset with 2.6m images, the processed dataset can be found in ghoskno/laion-art-en-colorcanny.

Training Details

• —Hardware: Google Cloud TPUv4-8 VM

• —Optimizer: AdamW

• —Train Batch Size: 4 x 4 = 16

• —Learning rate: 0.00001 constant

• —Gradient Accumulation Steps: 4

• —Resolution: 512

• —Train Steps: 36000