preran123/silhouette-studio

SILHOUETTE Studio

Minimalist Neural Pose Synthesis & Rapid Latent Consistency Engine Built with PyTorch, HuggingFace Diffusers, ControlNet (OpenPose), and LCM-LoRA on Stable Diffusion 1.5.

1. Executive Summary

SILHOUETTE Studio is a high-performance generative platform designed for real-time spatial pose conditioning and text-to-image synthesis.

Traditional diffusion pipelines require 30 to 50 denoising iterations per image, taking anywhere from 10 to 30 seconds on consumer GPUs. SILHOUETTE Studio fuses Latent Consistency Models (LCM-LoRA) directly into the base UNet's cross-attention layers, allowing the pipeline to predict clean latent destinations in 4 to 8 steps (~1.5 to 3.0 seconds) without compromising anatomical fidelity or structural pose alignment.

2. Core Capabilities

3. High-Level Architecture & Data Flow

[ User Input ] ──┬──► Text Prompt ──────────► [ CLIP Text Encoder ] ─────────► (1, 77, 768) Embedding
                 │
                 ├──► Reference Photo ──────► [ OpenPose Detector ] ──────┐
                 │                                                        │
                 └──► Canvas Sketch ──────────────────────────────────────┴──► (1, 3, 512, 512) Pose Tensor
                                                                                         │
                                                                                         ▼
[ Gaussian Noise Tensor ] ◄────────────────────────────────────────────────── [ ControlNet v1.1 ]
  z_T: (1, 4, 64, 64)                                                                    │
         │                                                                               │ Extracts 13 multi-scale
         ▼                                                                               │ residual feature maps
┌────────────────────────────────────────────────────────────────────────────────────────┴────────────────────┐
│                                     Iterative LCM Denoising Loop (4–8 Steps)                                │
│                                                                                                             │
│   z_t ──► [ UNet + Fused LCM-LoRA ] ◄── (ControlNet Residuals * CN_Scale) ──► Predicts Clean Latent z_0    │
│                 │                                                                                           │
│                 └──► [ LCMScheduler ] ──► Calculates Step z_(t-1) ──► (Repeats for N steps)                 │
└────────────────────────────────────────────────┬────────────────────────────────────────────────────────────┘
                                                 │
                                                 ▼
                                     Final Clean Latent z_0
                                       (1, 4, 64, 64)
                                                 │
                                                 ▼
                                     [ VAE Decoder (ft-mse) ]
                                                 │
                                                 ▼
                                     Output PIL RGB Image
                                       (1, 3, 512, 512)

4. Deep-Dive: How the Models Work Together

A. The Difference Between Standard Diffusion and LCM

• —Standard Diffusion (Velocity Prediction): Standard SD models solve a curved Ordinary Differential Equation (ODE) trajectory by taking 30–50 tiny incremental baby steps along the noise curve. If you try to jump 10 steps at once, the calculation veers off the manifold into distorted garbage.
• —Latent Consistency Model (Destination Prediction): LCM is parameterized with a consistency function $f(zt, t) \to z0$. Regardless of the current noise level $t$, the UNet is trained to predict the final clean image $z_0$ directly in a single leap. Running 4 steps simply repeats this prediction from successively clearer vantage points, refining fine details (eyes, skin texture, reflections).

B. Why Low Step Counts (4–8) are Mandatory with LCM

• —4–6 Steps (Sweet Spot): Optimal balance of contrast, sharpness, and prompt adherence.
• —20+ Steps (Degradation): Because LCM was distilled specifically for sparse leaps, running excessive steps causes small mathematical approximation errors ($~0.5\%$ per step) to compound and multiply, resulting in burnt contrast, oversaturation, and noise artifacts.

C. ControlNet Residual Injection

ControlNet creates a locked copy of the UNet's 12 encoder blocks and 1 middle block. It takes the $512 \times 512$ OpenPose skeleton image, encodes it through zero-convolution layers, and outputs 13 residual feature maps.

During each step of the denoising loop: $$\text{UNet\Layer\Input} = \text{UNet\Features} + (\alpha \cdot \text{ControlNet\Residual})$$ where $\alpha$ is the ControlNet Conditioning Scale (default 0.85). This guarantees that body posture, arm angles, and limb positions are strictly adhered to while allowing the base model full freedom to generate clothing, backgrounds, and artistic style.

5. Model Inventory

6. Deployment Guide: Where and How to Host

Because diffusion models require ~6 GB of weights and a GPU with at least 6 GB VRAM, traditional serverless providers (Vercel, AWS Lambda, Heroku) cannot host this directly. Below are the top 3 recommended deployment options:

Option A: Hugging Face Spaces (Recommended - Direct Git Push)

Hugging Face Spaces natively recognizes the YAML metadata at the top of this README.md.

1. 1.Create a free account at huggingface.co.
2. 2.Click New Space -> Name it silhouette-studio -> Select SDK: Gradio.
3. 3.Choose ZeroGPU (free dynamic A100 GPU) or T4 Small ($0.60/hr).
4. 4.Push this repository to your Space:

   git remote add space https://huggingface.co/spaces/<your-username>/silhouette-studio
   git push space main

1. 1.Hugging Face will parse the frontmatter, install requirements.txt, and automatically boot app.py.

Option B: Modal Labs (Serverless GPU - Cost Efficient)

Modal allows you to run PyTorch containers on serverless NVIDIA A10G/T4 GPUs with zero idle server cost.

Example `modal_app.py` wrapper:

import modal

app = modal.App("silhouette-studio")
image = (
    modal.Image.debian_slim(python_version="3.10")
    .pip_install_from_requirements("requirements.txt")
    .pip_install("torch", "torchvision", index_url="https://download.pytorch.org/whl/cu121")
)

@app.function(gpu="T4", image=image, container_idle_timeout=60)
@modal.web_server(port=7860)
def serve():
    from app import build_app
    demo = build_app()
    demo.launch(server_name="0.0.0.0", server_port=7860)

Deploy with:

modal deploy modal_app.py

Option C: RunPod / Vast.ai (Dedicated GPU - High Traffic)

If you need a dedicated instance running 24/7 for ~$0.20/hour:

1. 1.Deploy a RunPod Secure Cloud instance with an RTX 4000 Ada (20GB) or T4 (16GB).
2. 2.Select the runpod/pytorch:2.1.0-py3.10-cuda12.1.1-devel-ubuntu22.04 template.
3. 3.SSH into the pod:

   git clone https://github.com/your-username/real-aura.git
   cd real-aura
   pip install -r requirements.txt
   python app.py

1. 1.Connect via the provided HTTP proxy port 7860.

7. Local Setup & Installation

Prerequisites

• —Operating System: Windows 10/11, Linux, or macOS (Apple Silicon).
• —GPU: NVIDIA GPU with $\ge 6 \text{ GB VRAM}$ recommended.
• —System RAM: $\ge 16 \text{ GB}$.
• —Disk Space: $\approx 8 \text{ GB}$ free for model cache.

Step 1: Clone the Repository

git clone https://github.com/your-username/real-aura.git
cd real-aura

Step 2: Install PyTorch with CUDA

Install PyTorch compiled with your matching CUDA version (example for CUDA 12.1):

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121

Step 3: Install Project Dependencies

pip install -r requirements.txt

Step 4: Launch the Studio

python app.py

The application will launch and automatically open in your default browser at:

http://localhost:7860

Note on Initial Startup: On the very first generation, Diffusers downloads the base weights (~6 GB total) from Hugging Face Hub to ./models/. Subsequent boots load directly from cache in ~10 seconds.

8. Parameter Guide & Recommended Settings

9. Project Directory Structure

real-aura/
├── app.py                  # Gradio web application & monochrome UI styling
├── pipeline/
│   ├── __init__.py         # Package exports
│   ├── loader.py           # Singleton model loader (VAE, UNet, ControlNet, LCM-LoRA)
│   ├── processor.py        # OpenPose detection, sketch parsing & normalization
│   └── generate.py         # Core inference functions (generate_with_pose, generate_t2i)
├── utils/
│   ├── __init__.py         # Package exports
│   ├── canvas.py           # Skeleton canvas overlay & OpenPose joint color definitions
│   └── image_utils.py      # PIL <-> NumPy conversions, aspect ratio resizing & output saving
├── models/                 # Model cache directory (auto-created, gitignored)
├── outputs/                # Image archive directory (auto-created, gitignored)
├── requirements.txt        # Production dependency specifications
└── README.md               # Technical architecture & deployment documentation

10. License

This project is distributed under the Apache-2.0 License. Base model weights (stable-diffusion-v1-5, controlnet_openpose, lcm-lora) are subject to their respective OpenRAIL and CreativeML licenses.