RiddleHe/SD14_pathology_controlnet

<!-- This model card has been generated automatically according to the information the training script had access to. You should probably proofread and complete it, then remove this comment. -->

controlnet-RiddleHe/SD14pathologycontrolnet

These are controlnet weights trained on CompVis/stable-diffusion-v1-4 with new type of conditioning. You can find some example images below.

prompt: A histopathology image of breast cancer tissue. [image] prompt: An image of breast cancer histopathology with detailed cellular structures. [image]

Intended uses & limitations

How to use

controlnet = ControlNetModel.from_pretrained("RiddleHe/SD14_pathology_controlnet", torch_dtype=torch.float16)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "RiddleHe/SD14_pathology_base", controlnet=controlnet, torch_dtype=torch.float16
)
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
pipe.to('cuda')

prompt = "A histopathology image of breast cancer tissue."
mask = mask.convert("RGB")  # Provide a mask

generator = torch.Generator(device='cuda').manual_seed(42)

with torch.no_grad():
  out = pipe(prompt, image=mask, num_inference_steps=70, num_images_per_prompt=3, generator=generator).images

Limitations and bias

[TODO: provide examples of latent issues and potential remediations]

Training details

The model is trained on 28216 image-mask pairs from the BRCA breast cancer dataset. Input is mask and output is image.

Mask is a single channel image with integer values from 0 to 21 representing 22 classes, eg. 1 representing tumor, 2 representing stroma.