Oysiyl/controlnet-lora-brightness-flux

FLUX ControlNet LoRA - Brightness Control (10k @ 1024×1024)

A Control LoRA model trained on FLUX.1-dev to control image generation through brightness/grayscale information. This model uses LoRA (Low-Rank Adaptation) combined with ControlNet architecture for efficient control, providing an ultra-lightweight control mechanism for FLUX's powerful 12B parameter model.

Model Description

This Control LoRA enables brightness-based conditioning for FLUX image generation. By providing a grayscale image as input, you can control the brightness distribution and lighting structure while maintaining creative freedom through text prompts.

Key Features:

• —🎨 Excellent brightness and pattern control with FLUX's superior quality
• —🚀 Compact model size: ~685MB per checkpoint
• —⚡ Fast inference: Leverages FLUX's optimized architecture
• —💡 Fixed conditioning strength: Baked into LoRA weights during training
• —🔄 Compatible with Diffusers: Uses standard FluxControlPipeline
• —📦 Multiple checkpoints: Track training progression at 25%, 50%, 75%, 100%
• —🖼️ Native FLUX resolution: Trained at 1024×1024
• —🎯 XLabs-proven parameters: Validated hyperparameters for stability

Intended Uses:

• —Artistic QR code generation
• —Image recoloring and colorization
• —Lighting control in text-to-image generation
• —Brightness-based pattern integration
• —Watermark and subtle pattern embedding
• —Photo enhancement and stylization

Training Details

Training Data

Trained on 10,000 samples from latentcat/grayscale_image_aesthetic_3M:

• —High-quality aesthetic images
• —Paired with grayscale/brightness versions
• —Native resolution: 1024×1024

Training Configuration

XLabs Proven Parameters

This model uses hyperparameters validated by XLabs-AI for FLUX ControlNet training:

• —Learning Rate 2e-5: 5x lower than typical SDXL rates, critical for FLUX's 12B parameters
• —Warmup 10 steps: Gradual LR increase for training stability
• —Weight Decay 0.01: L2 regularization to prevent overfitting
• —Max Grad Norm 1.0: Gradient clipping to prevent explosion

These parameters provide stable training and prevent divergence in large models.

Model Size Comparison

Usage

Installation

pip install diffusers transformers accelerate torch

Basic Usage

import torch
import sys
sys.path.insert(0, '/path/to/diffusers/src')

from diffusers import FluxControlPipeline
from PIL import Image

# Load FLUX Control Pipeline
pipe = FluxControlPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16,
)

# Load Control LoRA weights
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-flux")
pipe.to("cuda")

# Load grayscale/brightness control image
control_image = Image.open("path/to/grayscale_image.png")
control_image = control_image.resize((1024, 1024))

# Generate image
prompt = "a beautiful garden scene with colorful flowers and butterflies, highly detailed, professional photography, vibrant colors"

image = pipe(
    prompt=prompt,
    control_image=control_image,
    num_inference_steps=28,  # FLUX default
    guidance_scale=3.5,       # FLUX default
    height=1024,
    width=1024,
).images[0]

image.save("output.png")

Adjusting Control Strength

You can control the LoRA conditioning strength using set_adapters() with adapter_weights:

# Load pipeline and LoRA
pipe = FluxControlPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16,
)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-flux")
pipe.to("cuda")

# Set adapter weight (scale) - controls conditioning strength
pipe.set_adapters(["default_0"], adapter_weights=[0.75])

# Now generate with adjusted strength
image = pipe(
    prompt=prompt,
    control_image=control_image,
    num_inference_steps=28,
    guidance_scale=3.5,
    height=1024,
    width=1024,
).images[0]

Recommended Scale Ranges:

• —0.5-0.7: Subtle control with hints of pattern
• —0.75-1.0: Balanced control (recommended for artistic QR codes)
• —1.0-1.25: Strong control with clear patterns
• —1.25-1.5: Very strong control

Note: Unlike SDXL ControlNet which uses controlnet_conditioning_scale, FLUX Control LoRA uses set_adapters() with adapter_weights to control conditioning strength.

Artistic QR Code Generation

import qrcode
from PIL import Image
import torch
from diffusers import FluxControlPipeline

# Generate QR code
qr = qrcode.QRCode(
    version=None,
    error_correction=qrcode.constants.ERROR_CORRECT_H,  # HIGH (30% tolerance)
    box_size=16,   # 16 pixels per module
    border=6,      # 6 modules border
)
qr.add_data("https://your-url.com")
qr.make(fit=True)

qr_image = qr.make_image(fill_color="black", back_color="white")
qr_image = qr_image.resize((1024, 1024), Image.LANCZOS).convert("RGB")

# Load pipeline with Control LoRA
pipe = FluxControlPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16,
)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-flux")
pipe.to("cuda")

# Generate artistic QR code
image = pipe(
    prompt="a beautiful garden with colorful flowers and butterflies, highly detailed, professional photography",
    control_image=qr_image,
    num_inference_steps=28,
    guidance_scale=3.5,
    height=1024,
    width=1024,
).images[0]

image.save("artistic_qr.png")

Using Different Checkpoints

The model includes intermediate checkpoints from throughout training:

# Early checkpoint (25% - 2,500 samples, step 312)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-flux",
                       subfolder="checkpoint-312")

# Mid checkpoint (50% - 5,000 samples, step 624)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-flux",
                       subfolder="checkpoint-624")

# Late checkpoint (75% - 7,500 samples, step 936)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-flux",
                       subfolder="checkpoint-936")

# Final model (10,000 samples, main branch - recommended)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-flux")

Scale Comparison Grids

Visual comparison of different adapter weights (scales) for each checkpoint. Each grid shows the same prompt and QR code at scales 0.5, 0.75, 1.0, 1.25, and 1.5.

Checkpoint-312 (25% trained, 2,500 samples)

[image]

Scale progression: From subtle (0.5) to very strong (1.5) control. Early checkpoint shows dense pattern integration with visible QR structure across all scales.

Checkpoint-624 (50% trained, 5,000 samples)

[image]

Scale progression: Improved artistic balance at mid-training. Natural garden scenes emerge with adjustable pattern strength.

Checkpoint-936 (75% trained, 7,500 samples)

[image]

Scale progression: Refined artistic quality. Excellent detail preservation with smooth scale transitions.

Final Model (100% trained, 10,000 samples)

[image]

Scale progression: Best overall quality. Recommended scales: 0.75-1.0 for artistic QR codes, 1.0-1.25 for pattern integration.

Scale Selection Guide

Based on the comparison grids:

Checkpoint Progression

All checkpoints are included on the main branch to show training progression:

Checkpoint-312 (25% trained, 2,500 samples)

<img src="examples/fluxcontrolnetlora_checkpoint-312.png" width="512" alt="Checkpoint 312">

Early training stage - dense pattern integration with strong QR visibility.

Checkpoint-624 (50% trained, 5,000 samples)

<img src="examples/fluxcontrolnetlora_checkpoint-624.png" width="512" alt="Checkpoint 624">

Mid-training - improved artistic balance with natural perspective and composition.

Checkpoint-936 (75% trained, 7,500 samples)

<img src="examples/fluxcontrolnetlora_checkpoint-936.png" width="512" alt="Checkpoint 936">

Advanced training - refined floral composition with excellent detail and artistic quality.

Final Model (100% trained, 10,000 samples)

<img src="examples/fluxcontrolnetlora_final.png" width="512" alt="Final Model">

Final model - unique artistic interpretation with frame effect and natural scene integration.

Original QR Code

<img src="examples/fluxcontrolnetloraqroriginal.png" width="256" alt="QR Original">

Validation QR code used during training (https://google.com, HIGH error correction).

Performance Comparison

vs SDXL Control LoRA

vs Full ControlNet

When to Use This Model

✅ Use This FLUX Control LoRA When:

• —Creating artistic images with FLUX's superior quality
• —Need brightness-based control for FLUX generation
• —Want production-ready artistic QR codes
• —Working with FLUX.1-dev as base model
• —Require checkpoint progression for analysis
• —Need efficient storage vs full ControlNet

⚠️ Consider Alternatives When:

• —Working with smaller datasets (SDXL trains faster with 10k samples)
• —Need minimal model size (SDXL Control LoRA is 24MB vs 685MB)
• —Require extremely precise control (use full ControlNet)
• —Need multiple control types - train separate LoRAs

Limitations

Current Limitations

• —FLUX-specific: Only works with FLUX.1-dev base model
• —Grayscale conditioning only: Trained specifically for brightness/grayscale control
• —Single control type: Only brightness, not other conditioning types (pose, depth, etc.)
• —Larger model size: 685MB vs SDXL's 24MB (includes more parameters)
• —Requires diffusers: Uses standard FluxControlPipeline from diffusers library

Recommendations

• —Use set_adapters() with adapter_weights=[0.75-1.0] for best artistic QR results
• —For multiple control types, train separate LoRAs
• —Use final model for best results (recommended)
• —Use checkpoint-936 (75%) for alternative artistic style
• —Experiment with different scales (0.5-1.5) to find the right balance

Training Script

#!/bin/bash
set -e

# Configuration
LOCAL_DATASET_10K="$HOME/.cache/huggingface/datasets_partial/grayscale_10k/combined"
OUTPUT_DIR="$HOME/controlnet-lora-brightness-flux-10k"
VALIDATION_QR="$HOME/validation_qr.png"
CHECKPOINT_STEPS=78  # Checkpoint every 2,500 samples

# Change to training directory
cd diffusers/examples/flux-control

# Training with XLabs proven parameters
accelerate launch --mixed_precision="bf16" train_control_lora_flux.py \
  --pretrained_model_name_or_path="black-forest-labs/FLUX.1-dev" \
  --dataset_name="$LOCAL_DATASET_10K" \
  --output_dir="$OUTPUT_DIR" \
  --mixed_precision="bf16" \
  --resolution=1024 \
  --learning_rate=2e-5 \
  --lr_scheduler=constant \
  --lr_warmup_steps=10 \
  --adam_weight_decay=0.01 \
  --max_grad_norm=1.0 \
  --proportion_empty_prompts=0.2 \
  --rank=128 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=8 \
  --num_train_epochs=1 \
  --gradient_checkpointing \
  --checkpointing_steps=$CHECKPOINT_STEPS \
  --validation_steps=$CHECKPOINT_STEPS \
  --validation_image="$VALIDATION_QR" \
  --validation_prompt="a beautiful garden scene with colorful flowers and butterflies, highly detailed, professional photography, vibrant colors" \
  --num_validation_images=4 \
  --guidance_scale=3.5 \
  --seed=42 \
  --dataloader_num_workers=4 \
  --report_to="wandb" \
  --tracker_project_name="controlnet-lora-brightness-flux-10k"

Available Checkpoints

All checkpoints are available in the main branch:

• —Root directory: Final model (10,000 samples, 1,250 steps, recommended)
• —checkpoint-312/: Early checkpoint (2,500 samples, 25% trained)
• —checkpoint-624/: Mid checkpoint (5,000 samples, 50% trained)
• —checkpoint-936/: Late checkpoint (7,500 samples, 75% trained)

Key Differences: FLUX vs SDXL

Troubleshooting

OOM (Out of Memory) Errors

If you encounter OOM during inference:

# Enable CPU offload
pipe.enable_model_cpu_offload()

# Or use sequential CPU offload (slower but less memory)
pipe.enable_sequential_cpu_offload()

Loading Errors

Make sure you have the latest diffusers:

pip install --upgrade diffusers transformers accelerate

Quality Issues

• —Use guidance_scale=3.5 (FLUX default)
• —Use num_inference_steps=28 or higher
• —Try different checkpoints (final model recommended)
• —Ensure control image is 1024×1024 resolution

Citation

@misc{controlnet-lora-brightness-flux,
  author = {Oysiyl},
  title = {FLUX ControlNet LoRA - Brightness Control (10k @ 1024×1024)},
  year = {2026},
  publisher = {HuggingFace},
  journal = {HuggingFace Model Hub},
  howpublished = {\url{https://huggingface.co/Oysiyl/controlnet-lora-brightness-flux}}
}

Acknowledgments

• —Built with 🤗 Diffusers
• —Base model: FLUX.1-dev by Black Forest Labs
• —XLabs proven parameters: x-flux by XLabs-AI
• —Dataset: grayscale_image_aesthetic_3M by latentcat
• —Training infrastructure: NVIDIA H100 80GB
• —LoRA implementation: PEFT by Hugging Face

License

Apache 2.0 License. The base FLUX.1-dev model has separate license terms at black-forest-labs/FLUX.1-dev.