goatom/Image_Edits

🎨 Zero-Shot Image Editing Assistant

Upload an image → Describe your edit → Get AI-generated results Powered by Stable Diffusion img2img + LoRA Transfer Learning

📋 Table of Contents

• —How It Works
• —Architecture
• —Transfer Learning Explained
• —Setup Instructions
• —GPU Configuration
• —Model Download
• —Running the App
• —Example Prompts
• —Docker Deployment
• —HuggingFace Spaces Deployment
• —Troubleshooting
• —Performance Optimization
• —Folder Structure

🧠 How It Works

1. 1.User uploads a source image (PNG/JPG/WebP)
2. 2.User writes a natural-language editing prompt
3. 3.Prompt Parser extracts style keywords, intensity modifiers & object edits
4. 4.Stable Diffusion img2img encodes the image into latent space, adds noise proportional to strength, and denoises guided by the text prompt
5. 5.LoRA adapter (transfer learning) modulates the model's style behaviour
6. 6.Result is displayed side-by-side with the original and can be downloaded

🏗 Architecture

Browser (HTML/CSS/JS)
    │
    ├── POST /upload     →  Save image to static/uploads/
    ├── POST /generate   →  Prompt Parser → Diffusion Pipeline → Output
    └── GET  /download   →  Serve output file
          │
   Flask Server (app.py)
          │
          ├── utils/prompt_parser.py      ← NLP-style prompt analysis
          └── utils/diffusion_pipeline.py ← SD img2img + LoRA
                    │
             HuggingFace Diffusers
                    │
         runwayml/stable-diffusion-v1-5 (pre-trained)

🔬 Transfer Learning Explained

This project uses two levels of transfer learning — neither trains a model from scratch:

Level 1: Pre-Trained Stable Diffusion

The base model (runwayml/stable-diffusion-v1-5) was trained on LAION-5B (~5 billion image-text pairs). We load these pre-trained weights directly — this is the simplest form of transfer learning: feature reuse.

pipe = StableDiffusionImg2ImgPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16  # Half-precision for GPU efficiency
)
pipe = pipe.to("cuda")  # GPU acceleration

Level 2: LoRA (Low-Rank Adaptation)

LoRA is a parameter-efficient fine-tuning technique:

• —The base model's attention weight matrices W are frozen (not changed)
• —Two small matrices A (down-projection) and B (up-projection) are added: W' = W + scale × (B @ A)
• —LoRA weights are only a few MB vs. the full model's ~4 GB
• —They encode domain-specific knowledge (e.g., anime style, oil painting style)
• —We load pre-trained LoRA weights from HuggingFace Hub: pipe.load_lora_weights("lora-library/some-style")

Why this is transfer learning:

• —The base model's knowledge is transferred to new tasks
• —LoRA adapts the model to a new domain without retraining the full model
• —Inference is fast because only the small adapter weights change the model's behaviour

🚀 Setup Instructions

Prerequisites

• —Python 3.10+
• —8 GB+ RAM (for CPU mode)
• —NVIDIA GPU with 4+ GB VRAM (recommended, not required)
• —Git installed

Step 1: Clone / Navigate to the project

cd c:\Users\tomge\Desktop\ai_img

Step 2: Create a virtual environment (recommended)

python -m venv venv

# Windows:
venv\Scripts\activate

# Linux/Mac:
source venv/bin/activate

Step 3: Install dependencies

pip install -r requirements.txt

Step 4: Install PyTorch with CUDA (if you have an NVIDIA GPU)

# Check your CUDA version first:
nvidia-smi

# Install matching PyTorch:
# CUDA 11.8:
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118

# CUDA 12.1:
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121

# CPU only (no GPU):
pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu

Q

🖥 GPU Configuration

Check GPU availability

import torch
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"GPU: {torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'None'}")
print(f"VRAM: {torch.cuda.get_device_properties(0).total_mem / 1e9:.1f} GB" if torch.cuda.is_available() else "")

Memory optimization (already enabled in the code)

These are all enabled by default to keep your laptop cool.

📥 Model Download

The model downloads automatically on the first generation request. No manual download needed.

Models are cached in the models/ directory. Subsequent runs use the cache.

Manual download (optional)

from diffusers import StableDiffusionImg2ImgPipeline
StableDiffusionImg2ImgPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", cache_dir="./models")

▶️ Running the App

python app.py

Open your browser at http://localhost:5000

💡 Example Prompts

🐳 Docker Deployment

Build and run

docker build -t zero-shot-editor .
docker run -p 5000:5000 zero-shot-editor

With GPU (NVIDIA Container Toolkit required)

docker run --gpus all -p 5000:5000 zero-shot-editor

🤗 HuggingFace Spaces Deployment

1. 1.Create a new Space at huggingface.co/spaces
2. 2.Select Gradio or Docker SDK
3. 3.Upload all project files
4. 4.Set Space hardware to T4 GPU (free tier available)
5. 5.The app will auto-build and deploy

For HuggingFace Spaces, you may need to change app.py's host:

app.run(host="0.0.0.0", port=7860)  # Spaces uses port 7860

🔧 Troubleshooting

⚡ Performance Optimization

Speed improvements

1. 1.Use GPU — 10-20x faster than CPU
2. 2.Reduce steps — 15-20 steps gives decent quality much faster
3. 3.Reduce image size — 384px instead of 512px
4. 4.Use float16 — already enabled for GPU

Memory savings

1. 1.Attention slicing — already enabled
2. 2.VAE slicing — already enabled
3. 3.CPU offload — set ENABLE_CPU_OFFLOAD = True in config.py for very low VRAM GPUs
4. 4.Reduce batch size — this app processes one image at a time (optimal)

Quality tips

1. 1.Increase steps to 40-50 for higher quality
2. 2.Guidance scale of 7-9 is usually optimal
3. 3.Strength of 0.4-0.6 preserves structure well
4. 4.Negative prompts are auto-generated by the prompt parser

📁 Folder Structure

zero_shot_editor/
│
├── app.py                          # Flask server (routes, API)
├── config.py                       # All configuration in one place
├── requirements.txt                # Python dependencies
├── Dockerfile                      # Container deployment
├── .dockerignore                   # Docker build exclusions
├── README.md                       # This file
│
├── models/                         # Cached model weights (auto-created)
│
├── static/
│   ├── css/
│   │   └── style.css               # Dark glassmorphism UI
│   ├── js/
│   │   └── app.js                  # Frontend logic
│   ├── uploads/                    # User-uploaded images
│   └── outputs/                    # AI-generated outputs
│
├── templates/
│   └── index.html                  # Main web interface
│
└── utils/
    ├── __init__.py
    ├── diffusion_pipeline.py       # SD img2img + LoRA pipeline
    └── prompt_parser.py            # NLP-style prompt analysis

📄 License

MIT License — Use freely for academic and personal projects.