mlpc-lab/TokenCompose_SD14_B

1---2license: creativeml-openrail-m3language:4- en5library_name: diffusers6pipeline_tag: text-to-image7tags:8- stable-diffusion9- cvpr10- text-to-image11- image-generation12- compositionality13---14# 🧩 TokenCompose SD14 Model Card15## 🎬CVPR 202416[TokenCompose_SD14_B](https://mlpc-ucsd.github.io/TokenCompose/) is a [latent text-to-image diffusion model](https://arxiv.org/abs/2112.10752) finetuned from the [**Stable-Diffusion-v1-4**](https://huggingface.co/CompVis/stable-diffusion-v1-4) checkpoint at resolution 512x512 on the [VSR](https://github.com/cambridgeltl/visual-spatial-reasoning) split of [COCO image-caption pairs](https://cocodataset.org/#download) for 24,000 steps with a learning rate of 5e-6. The training objective involves token-level grounding terms in addition to denoising loss for enhanced multi-category instance composition and photorealism. The "_A/B" postfix indicates different finetuning runs of the model using the same above configurations.17 18# 📄 Paper19 20Please follow [this](https://arxiv.org/abs/2312.03626) link.21 22# 🧨Example Usage23 24We strongly recommend using the [🤗Diffuser](https://github.com/huggingface/diffusers) library to run our model.25 26```python27import torch28from diffusers import StableDiffusionPipeline29 30model_id = "mlpc-lab/TokenCompose_SD14_B"31device = "cuda"32 33pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float32)34pipe = pipe.to(device)35 36prompt = "A cat and a wine glass"37image = pipe(prompt).images[0]  38    39image.save("cat_and_wine_glass.png")40```41 42# ⬆️Improvements over SD1443 44<table>45 46  <tr>47    <th rowspan="3" align="center">Method</th>48    <th colspan="9" align="center">Multi-category Instance Composition</th>49    <th colspan="2" align="center">Photorealism</th>50    <th colspan="1" align="center">Efficiency</th>51  </tr>52 53  <tr>54    <!-- <th align="center">&nbsp;</th> -->55    <th rowspan="2" align="center">Object Accuracy</th>56    <th colspan="4" align="center">COCO</th>57    <th colspan="4" align="center">ADE20K</th>58    <th rowspan="2" align="center">FID (COCO)</th>59    <th rowspan="2" align="center">FID (Flickr30K)</th>60    <th rowspan="2" align="center">Latency</th>61  </tr>62 63  <tr>64    <!-- <th align="center">&nbsp;</th> -->65    <th align="center">MG2</th>66    <th align="center">MG3</th>67    <th align="center">MG4</th>68    <th align="center">MG5</th>69    <th align="center">MG2</th>70    <th align="center">MG3</th>71    <th align="center">MG4</th>72    <th align="center">MG5</th>73  </tr>74 75  <tr>76    <td align="center"><a href="https://huggingface.co/CompVis/stable-diffusion-v1-4">SD 1.4</a></td>77    <td align="center">29.86</td>78    <td align="center">90.72<sub>1.33</sub></td>79    <td align="center">50.74<sub>0.89</sub></td>80    <td align="center">11.68<sub>0.45</sub></td>81    <td align="center">0.88<sub>0.21</sub></td>82    <td align="center">89.81<sub>0.40</sub></td>83    <td align="center">53.96<sub>1.14</sub></td>84    <td align="center">16.52<sub>1.13</sub></td>85    <td align="center">1.89<sub>0.34</sub></td>86    <td align="center"><u>20.88</u></td>87    <td align="center"><u>71.46</u></td>88    <td align="center"><b>7.54</b><sub>0.17</sub></td>89  </tr>90 91  <tr>92    <td align="center"><a href="https://github.com/mlpc-ucsd/TokenCompose"><strong>TokenCompose (Ours)</strong></a></td>93    <td align="center"><b>52.15</b></td>94    <td align="center"><b>98.08</b><sub>0.40</sub></td>95    <td align="center"><b>76.16</b><sub>1.04</sub></td>96    <td align="center"><b>28.81</b><sub>0.95</sub></td>97    <td align="center"><u>3.28</u><sub>0.48</sub></td>98    <td align="center"><b>97.75</b><sub>0.34</sub></td>99    <td align="center"><b>76.93</b><sub>1.09</sub></td>100    <td align="center"><b>33.92</b><sub>1.47</sub></td>101    <td align="center"><b>6.21</b><sub>0.62</sub></td>102    <td align="center"><b>20.19</b></td>103    <td align="center"><b>71.13</b></td>104    <td align="center"><b>7.56</b><sub>0.14</sub></td>105  </tr>106 107</table>108 109# 📰 Citation110```bibtex111@InProceedings{Wang2024TokenCompose,112    author    = {Wang, Zirui and Sha, Zhizhou and Ding, Zheng and Wang, Yilin and Tu, Zhuowen},113    title     = {TokenCompose: Text-to-Image Diffusion with Token-level Supervision},114    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},115    month     = {June},116    year      = {2024},117    pages     = {8553-8564}118}119```