mediasynthesismuseum/aligndraw

AlignDRAW (resurrected) — one of the first text-to-image models

A trained-from-scratch PyTorch resurrection of AlignDRAW (Mansimov, Parisotto, Ba, Salakhutdinov — Generating Images from Captions with Attention, ICLR 2016), one of the earliest neural text-to-image models. The original Theano code shipped no pretrained weights and the framework is long dead — so this model was re-implemented in PyTorch and trained on the original still-online COCO 32×32 caption data (cs.toronto.edu/~emansim), for `mediasynthesismuseum/aligndraw`.

It's a recurrent VAE that draws an image onto a canvas over 32 steps, attending to the caption words at each step (Bahdanau alignment) with a learned sequential prior. Output is a blurry 32×32 blob whose colour/layout tracks the caption — exactly the fidelity the 2015 model had.

Files

• —aligndraw_coco.pt — trained weights (33M params; dims match the original coco-captions-32x32).
• —model.py — the faithful PyTorch AlignDRAW definition.
• —dictionary.pkl — the original 25,322-word caption dictionary (word → index; includes UNK).

Usage

import torch, pickle
from model import AlignDRAW
m = AlignDRAW(); m.load_state_dict(torch.load("aligndraw_coco.pt", map_location="cpu")); m.eval()
d = pickle.load(open("dictionary.pkl","rb"), encoding="latin1")
toks = [d.get(w, d["UNK"]) for w in "a red stop sign on a street corner .".split()]
y = torch.tensor([toks]).expand(8, -1)
imgs = m.generate(y)  # (8, 3, 32, 32) in [0,1]

Credit: Elman Mansimov, Emilio Parisotto, Jimmy Ba, Ruslan Salakhutdinov (original AlignDRAW); built on DRAW (Gregor et al.).