Hadimeeee/pixel-art-lora-sdxl

Pixel Art LoRA — Stuffed Animal Sprite Converter

SDXL DreamBooth LoRA trained to convert stuffed animal photos into 8-bit pixel art sprites.

Developed as part of the 몽글마을 (Mongle Village) project — an AI-based stuffed animal persona app.

What it does

Converts a photo of a stuffed animal into a 16x16-style pixel art character sprite with white background, bold outlines, and flat colors.

Input photo → Background removal (rembg) → Edge detection (ControlNet) → Pixel art (this LoRA)

Model Details

Performance (20 test images, vs 6 other models)

How to use

Intended deployment

This repository is designed to be used as a HuggingFace Hub package for a RunPod GPU server.

RunPod server
  -> download this HuggingFace repo
  -> load pipeline.py
  -> run rembg + Canny + ControlNet + SDXL + LoRA + quantization
  -> expose the result through an API

HuggingFace stores the LoRA weights and pipeline code. The actual inference runs on RunPod.

Requirements

pip install -r requirements.txt

Download from HuggingFace and run locally/on RunPod

from huggingface_hub import snapshot_download
from PIL import Image

repo_dir = snapshot_download("Hadimeeee/pixel-art-lora-sdxl")

import sys
sys.path.insert(0, repo_dir)

from pipeline import load_pipeline

pipe = load_pipeline(repo_dir)
image = Image.open("your_image.jpg").convert("RGB")
result = pipe(image)["image"]
result.save("pixel_art_result.png")

RunPod serverless handler

Use runpod_handler.py as the serverless entrypoint. The handler expects a base64-encoded image:

{
  "input": {
    "image": "<base64 png or jpeg>",
    "num_inference_steps": 50,
    "guidance_scale": 7.5,
    "controlnet_conditioning_scale": 0.8,
    "strength": 0.75,
    "quantize": true,
    "n_colors": 32
  }
}

The response returns a base64-encoded PNG:

{
  "image": "<base64 png>",
  "rembg_ok": true
}

Pipeline breakdown

Note: rembg and ControlNet are not included in this file. They are separate open-source tools loaded at inference time.

Tips

• —Works best on stuffed animals and character-shaped objects with clear silhouettes
• —If background removal fails, the pipeline automatically falls back to the original image
• —For more pixel-art-like results, apply color quantization after generation:

  result.quantize(colors=32, method=Image.Quantize.MEDIANCUT, dither=Image.Dither.NONE).convert("RGB")