Modelpublic

cyberagent/ldm-vq-f16-rgba

sourceHugging Faceapache-2.0updated 1y agoView on Hugging Face

0likes218downloads

Model Card

Image Tokenizer (VQ Model) for RGBA Images

![arxiv paper](https://arxiv.org/abs/2409.19051) <a href='https://cyberagentailab.github.io/MarkupDM/'><img src='https://img.shields.io/badge/Project-Page-Green'></a>

</div>

[image]

This repository provides a specialized image tokenizer (VQ model) for RGBA images. The model can effectively compress RGBA images into discrete latent codes, which can be used for various downstream tasks such as graphic design completion, as demonstrated in our paper Multimodal Markup Document Models for Graphic Design Completion.

Model Details

The model is based on the image tokenizer from the latent diffusion model. We modified the original model (models/first_stage_models/vq-f16) to handle RGBA images by changing the input and output channels from 3 to 4. See our paper for more details.

Requirements

bash

cat <<EOF > requirements.txt
transformers>=4.42.4
omegaconf>=2.3.0
einops>=0.8.0
pillow>=10.4.0
pytorch_lightning<2.0.0
huggingface-hub>=0.24.0
git+https://github.com/ktrk115/latent-diffusion.git@23d5a49
git+https://github.com/illeatmyhat/taming-transformers.git@aeabaa3
EOF

bash

pip install -r requirements.txt

Usage

python

import torch
from PIL import Image
from transformers import AutoImageProcessor, AutoModel

image_processor = AutoImageProcessor.from_pretrained("cyberagent/ldm-vq-f16-rgba", trust_remote_code=True)
model = AutoModel.from_pretrained("cyberagent/ldm-vq-f16-rgba", trust_remote_code=True)

# Image reconstruction
img = Image.open("path/to/image.png")
example = image_processor(img)
with torch.inference_mode():
    recon, _ = model.model(example["image"].unsqueeze(0))
recon_img = image_processor.postprocess(recon[0])
recon_img.save("recon.png")

License

This repository is released under the Apache-2.0 license.

Citation

bibtex

@inproceedings{Kikuchi2025,
  title     = {Multimodal Markup Document Models for Graphic Design Completion},
  author    = {Kotaro Kikuchi and Ukyo Honda and Naoto Inoue and Mayu Otani and Edgar Simo-Serra and Kota Yamaguchi},
  booktitle = {ACM International Conference on Multimedia},
  year      = {2025},
  doi       = {10.1145/3746027.3755420}
}