mayank-dubey-ai/sdxl-activation-steering

SDXL Image Activation Steering: Continuous Concept Trajectory Control

This repository demonstrates Representation Engineering and Activation Steering on Diffusion Models (SDXL Turbo) using PyTorch and Hugging Face diffusers on an NVIDIA H100 GPU.

⚡ How It Works

Instead of retraining a LoRA or using binary negative prompt weights, we extract continuous concept steering vectors:

$$\mathbf{v}{\text{style}} = \frac{1}{N} \sum{i=1}^N \left( \mathbf{e}{\text{cyberpunk\neon}}^{(i)} - \mathbf{e}{\text{golden\hour}}^{(i)} \right)$$

During the diffusion denoising trajectory, we steer the latent embeddings continuously:

$$\mathbf{e}{\text{steered}} = \mathbf{e}{\text{neutral}} + \alpha \cdot \mathbf{v}_{\text{style}}$$

• —$\alpha = -1.2$ (Golden Hour): Continuous shift into warm, natural golden hour sunlight and organic daytime scenery.
• —$\alpha = 0.0$ (Neutral Baseline): Unsteered baseline output at fixed seed.
• —$\alpha = +1.2$ (Cyberpunk Neon): Continuous shift into dark cyberpunk nightscapes, glowing electric cyan/magenta reflections, and volumetric haze.

📁 Repository Structure

sdxl-activation-steering/
├── concept_pairs.json                     # Contrast prompts (Cyberpunk vs. Golden Hour)
├── extract_image_steering_vectors.py      # Computes sequence & pooled steering vectors
├── steer_image_generation.py              # Denoising inference loop with alpha sweeps
├── vectors/
│   └── cyberpunk_vs_goldenhour.pt         # Pre-computed SDXL steering vectors
├── outputs/                               # Individual steered outputs at each alpha
└── assets/
    └── image_steering_sweep_grid.png      # 5x3 side-by-side comparison grid

🚀 Reproduction

# 1. Clone repository
git clone https://huggingface.co/datasets/mayank-dubey-ai/sdxl-activation-steering
cd sdxl-activation-steering

# 2. Extract vectors
python3 extract_image_steering_vectors.py

# 3. Run image steering sweep
python3 steer_image_generation.py