GeroldMeisinger/controlnet-channels

Restore missing RGB channels

Restore a missing channel of a RGB image by using ControlNet to guide image generation of Stable Diffusion to infer missing channel from the other two channels.

• —See accompanying discussion at github.com - Channels RGB with detailed report and evaluations.
• —To restore images with missing channels you can use this space.
• —For evaluation images see the corresponding .zip's at "files".
• —To run your own evaluations you can use this script at gitlab.com.

Training

accelerate launch train_controlnet.py \
  --pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5" \
  --train_batch_size=4 \
  --gradient_accumulation_steps=8 \
  --proportion_empty_prompts=0.5
  --mixed_precision="fp16" \
  --learning_rate=1e-5 \
  --enable_xformers_memory_efficient_attention \
  --use_8bit_adam \
  --set_grads_to_none \
  --seed=0 \
  --num_train_epochs=2

Image dataset

• —laion2B-en aesthetics>=6.5 dataset
• —--minimagesize 512 --maxaspectratio 2 --resizemode="centercrop" --image_size 512
• —Cleaned with fastdup default settings
• —Data augmented with right-left flipped images
• —Resulting in 214244 images
• —Set whole channel to 0 by alternating between R-G-B channels