sayakpaul/trained-lumina2-lora-yarn

Lumina2 DreamBooth LoRA - trained-lumina2-lora-yarn

<Gallery />

Model description

These are trained-lumina2-lora-yarn DreamBooth LoRA weights for Alpha-VLLM/Lumina-Image-2.0.

The weights were trained using DreamBooth with the Lumina2 diffusers trainer.

Trigger words

You should use yarn art style to trigger the image generation.

The following system_prompt was also used used during training (ignore if None): None.

Download model

Download the *.safetensors LoRA in the Files & versions tab.

Use it with the 🧨 diffusers library

import torch
from diffusers import Lumina2Text2ImgPipeline

pipe = Lumina2Text2ImgPipeline.from_pretrained(
    "Alpha-VLLM/Lumina-Image-2.0", torch_dtype=torch.bfloat16
).to("cuda")

pipe.load_lora_weights("trained-lumina2-lora-yarn")
prompt = "a puppy in a pond, yarn art style"

image = pipe(
    prompt, 
    negative_prompt="bad quality, worse quality, degenerate quality", 
    guidance_scale=6,
    num_inference_steps=35, 
    generator=torch.manual_seed(0)
).images[0]

For more details, including weighting, merging and fusing LoRAs, check the documentation on loading LoRAs in diffusers.

Results

The model benefits from system_prompt. Here is a comparison across different system prompts:

<table> <thead> <tr> <th>No system prompt</th> <th>"Dark surrounding"<br>system prompt</th> <th>"Sunny surrounding"<br>system prompt</th> </tr> </thead> <tbody> <tr> <td><img src="yarnlora.png" alt="No system prompt image" width="200"></td> <td><img src="yarnloraYouareanassistantdesignedtogeneratesuperiorimageswithadarkoveralltheme.png" alt="Dark surrounding image" width="200"></td> <td><img src="yarnloraYouareanassistantdesignedtogeneratesuperiorimageswithabrightandshinyoverall_.png" alt="Sunny surrounding image" width="200"></td> </tr> <tr> <td colspan="3"> <div style="text-align: center; font-weight: bold;"> <b>Original prompt</b>: <i>a puppy in a pond, yarn art style</i> </div> </td> </tr> </tbody> </table>

<details> <summary>Code</summary>

import torch
from diffusers import Lumina2Text2ImgPipeline

pipe = Lumina2Text2ImgPipeline.from_pretrained(
    "Alpha-VLLM/Lumina-Image-2.0", torch_dtype=torch.bfloat16
).to("cuda")


system_prompts = [
    None, 
    "You are an assistant designed to generate superior images with a dark overall theme.",
    "You are an assistant designed to generate superior images with a bright and shiny overall theme."
]

pipe.load_lora_weights("trained-lumina2-lora-yarn")
prompt = "a puppy in a pond, yarn art style"

for sp in system_prompts:
    filename = "yarn_lora"
    image = pipe(
        prompt, 
        negative_prompt="bad quality, worse quality, degenerate quality",
        system_prompt=sp, 
        guidance_scale=6,
        num_inference_steps=35, 
        generator=torch.manual_seed(0)
    ).images[0]
    if sp:
        filename += "_" + "_".join(sp.split(" ")).replace(",", "").replace(".", "")
        filename = filename[:100]
    
    image.save(f"{filename}.png")

</details>