adrianbacon/realistic-universal-synthetic-small-0.0.0

Realistic Universal Synthetic Small v0.0.0

This is a small modernized MMDiT-x based model (modeled after the SD3.5 archtecture) meant to replace older unet based models like stable diffusion 1.4 and 1.5.

Currently in training.

Model Details

• —MMDiT (multi-modal diffusion transformer architecture)
• —Attention head width of 128, with 16 attention heads.
• —6 double stream transformers, 18 single stream transformers for a total of 24 layers. Approximately 1.6 billion parameters.
• —Modern 16 channel VAE.
• —A single modern natural language transformer based text encoder (Google T5-XXL)
• —Meant to run on a single 24GB GPU, or on a single 12GB GPU with diffusers sequential cpu offloading enabled, or split up between two 12GB GPUs.
• —Native 512x512, 512x768 (portrait), and 1024x512 (landscape) resolutions. Looking to be centered on just a couple of master resolutions that people can to use to crop to other common aspect ratios after the fact.

Model Purpose

This model (I hope) will ultimately be part of a larger community ecosystem where there is a small base model that can have have specialized fine tunes applied to it, and one or more larger models (a Medium, and/or Large/Extra Large version) with more parameters, but trained on the same data but at higher resolutions used to get higher resolution generations.

This smaller initial model is meant to allow fast iteration of prompts and seeds, then once a desired combination is arrived upon, the small generated image, prompt, and seed are taken to a larger model to generate the same image, just at a higher resolution. Because the larger model was trained on essentially the same data, but at a higher resolution, it should provide dramatically better higher resolution versions of a given prompt and seed.