gmmeyer/sprite-gpt

sprite-gpt

Small pixel-space diffusion models for emoji and game sprites, trained with rectified flow (flow matching) on a single RTX 5090. Code: github.com/gmmeyer/sprite-diffusion

The text model composes attributes not seen together in training — prompts below are "a purple dragon emoji", "an angry ghost sprite", "a blue cat face emoji", "a green skull sprite", "ice dungeon wall tile", "a red sword item", "a crying robot face emoji", "a yellow snake monster" (8 seeds each):

[image]

Both are ADM-style U-Nets (scale-shift GroupNorm conditioning, self-attention at 16x16 and 8x8) trained with the rectified-flow objective (x_t = (1-t)·x0 + t·ε, predict v = ε − x0, logit-normal timestep sampling). Weights are fp32 EMA (decay 0.9995). Sampling: 50-step Euler ODE; the conditional model uses classifier-free guidance (null class = index 31, condition dropout 10% at train time; guidance 3 works well).

[image]

Usage

Text-to-image (easiest via the repo CLI):

git clone https://github.com/gmmeyer/sprite-diffusion && cd sprite-diffusion && uv sync
uv run python -m spritegpt.sample --ckpt sprite-gpt-text64.pt \
    --prompt "a blue ghost sprite" --guidance 5 --out ghost.png

(sample.py accepts these exported files as well as training checkpoints.)

Class-conditional model, by hand:

# git clone https://github.com/gmmeyer/sprite-diffusion && cd sprite-diffusion && uv sync
import torch
from spritegpt.unet import UNet
from spritegpt.flow import sample
from spritegpt.utils import save_image_grid

ck = torch.load("sprite-gpt-cond64.pt", map_location="cuda", weights_only=False)
cfg = ck["config"]
model = UNet(
    img_size=cfg["img_size"],
    base=cfg["base"],
    ch_mult=tuple(int(c) for c in cfg["ch_mult"].split(",")),
    num_res=cfg["num_res"],
    attn_res=tuple(int(r) for r in cfg["attn_res"].split(",")),
    num_classes=cfg["num_classes"],
).cuda()
model.load_state_dict(ck["ema_state_dict"])
model.eval()

y = torch.full((16,), 24, device="cuda", dtype=torch.long)  # sprite/dcss-mon
with torch.autocast("cuda", dtype=torch.bfloat16):
    imgs = sample(model, 16, 64, steps=50, y=y, guidance=3.0,
                  num_classes=cfg["num_classes"], device="cuda")
save_image_grid(imgs.float(), "monsters.png", nrow=4)

Classes (conditional model)

Class ids are the alphabetical index into:

emoji/activities, emoji/animals-nature, emoji/component, emoji/extras-openmoji, emoji/extras-unicode, emoji/flags, emoji/food-drink, emoji/objects, emoji/people-body, emoji/smileys-emotion, emoji/symbols, emoji/travel-places, emoji/unknown, sprite/dawnlike, sprite/dcss-dungeon, sprite/dcss-effect, sprite/dcss-item, sprite/dcss-misc, sprite/dcss-mon, sprite/dcss-player, sprite/kenney-1bit, sprite/kenney-caves, sprite/kenney-characters, sprite/kenney-micro-roguelike, sprite/kenney-modern-city, sprite/kenney-pixel-platformer, sprite/kenney-pixel-shmup, sprite/kenney-rpg, sprite/kenney-tiny-battle, sprite/kenney-tiny-dungeon, sprite/kenney-tiny-town

Training data and attribution

Images composited onto white backgrounds at 32/64px:

• —OpenMoji — CC-BY-SA 4.0 (the reason this model card is CC-BY-SA)
• —Twemoji — CC-BY 4.0, © Twitter/X and contributors
• —Dungeon Crawl Stone Soup tiles — public domain
• —Kenney asset packs — CC0
• —DawnLike tileset — CC-BY 4.0, credit DragonDePlatino and DawnBringer

Limitations

• —The unconditional 32px emoji model partially memorizes its 8.5k-image training set (~1/3 of samples are near-duplicates of training images).
• —Rare classes (a few hundred images) produce less diverse samples.
• —sprite/kenney-characters is trained on paper-doll layers (bodies, clothing, hats as separate sprites), so it generates those parts rather than assembled characters, and retains some flat-green artifacts from a data-contamination issue fixed partway through training (see repo history).
• —Outputs are white-background RGB; no alpha channel.