google/pix2struct-ai2d-large

1---2language:3  - en4  - fr5  - ro6  - de7  - multilingual8pipeline_tag: visual-question-answering9inference: false10license: apache-2.011---12 13 14# Model card for Pix2Struct - Finetuned on AI2D (scientific diagram VQA) - large version15 16![model_image](https://s3.amazonaws.com/moonup/production/uploads/1678713353867-62441d1d9fdefb55a0b7d12c.png)17 18#  Table of Contents19 200. [TL;DR](#TL;DR)211. [Using the model](#using-the-model)222. [Contribution](#contribution)233. [Citation](#citation)24 25# TL;DR26 27Pix2Struct is an image encoder - text decoder model that is trained on image-text pairs for various tasks, including image captionning and visual question answering. The full list of available models can be found on the Table 1 of the paper:28 29![Table 1 - paper](https://s3.amazonaws.com/moonup/production/uploads/1678712985040-62441d1d9fdefb55a0b7d12c.png)30 31 32The abstract of the model states that: 33> Visually-situated language is ubiquitous—sources range from textbooks with diagrams to web pages with images and tables, to mobile apps with buttons and34forms. Perhaps due to this diversity, previous work has typically relied on domainspecific recipes with limited sharing of the underlying data, model architectures,35and objectives. We present Pix2Struct, a pretrained image-to-text model for36purely visual language understanding, which can be finetuned on tasks containing visually-situated language. Pix2Struct is pretrained by learning to parse37masked screenshots of web pages into simplified HTML. The web, with its richness of visual elements cleanly reflected in the HTML structure, provides a large38source of pretraining data well suited to the diversity of downstream tasks. Intuitively, this objective subsumes common pretraining signals such as OCR, language modeling, image captioning. In addition to the novel pretraining strategy,39we introduce a variable-resolution input representation and a more flexible integration of language and vision inputs, where language prompts such as questions40are rendered directly on top of the input image. For the first time, we show that a41single pretrained model can achieve state-of-the-art results in six out of nine tasks42across four domains: documents, illustrations, user interfaces, and natural images.43 44# Using the model 45 46This model has been fine-tuned on VQA, you need to provide a question in a specific format, ideally in the format of a Choices question answering47 48## Converting from T5x to huggingface49 50You can use the [`convert_pix2struct_checkpoint_to_pytorch.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/pix2struct/convert_pix2struct_checkpoint_to_pytorch.py) script as follows:51```bash52python convert_pix2struct_checkpoint_to_pytorch.py --t5x_checkpoint_path PATH_TO_T5X_CHECKPOINTS --pytorch_dump_path PATH_TO_SAVE --is_vqa53```54if you are converting a large model, run:55```bash56python convert_pix2struct_checkpoint_to_pytorch.py --t5x_checkpoint_path PATH_TO_T5X_CHECKPOINTS --pytorch_dump_path PATH_TO_SAVE --use-large --is_vqa57```58Once saved, you can push your converted model with the following snippet:59```python60from transformers import Pix2StructForConditionalGeneration, Pix2StructProcessor61 62model = Pix2StructForConditionalGeneration.from_pretrained(PATH_TO_SAVE)63processor = Pix2StructProcessor.from_pretrained(PATH_TO_SAVE)64 65model.push_to_hub("USERNAME/MODEL_NAME")66processor.push_to_hub("USERNAME/MODEL_NAME")67```68 69## Running the model70 71### In full precision, on CPU:72 73You can run the model in full precision on CPU:74```python75import requests76from PIL import Image77from transformers import Pix2StructForConditionalGeneration, Pix2StructProcessor78 79image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg"80image = Image.open(requests.get(image_url, stream=True).raw)81 82model = Pix2StructForConditionalGeneration.from_pretrained("google/pix2struct-ai2d-large")83processor = Pix2StructProcessor.from_pretrained("google/pix2struct-ai2d-large")84 85question = "What does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud"86 87inputs = processor(images=image, text=question, return_tensors="pt")88 89predictions = model.generate(**inputs)90print(processor.decode(predictions[0], skip_special_tokens=True))91>>> ash cloud92```93 94### In full precision, on GPU:95 96You can run the model in full precision on CPU:97```python98import requests99from PIL import Image100from transformers import Pix2StructForConditionalGeneration, Pix2StructProcessor101 102image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg"103image = Image.open(requests.get(image_url, stream=True).raw)104 105model = Pix2StructForConditionalGeneration.from_pretrained("google/pix2struct-ai2d-large").to("cuda")106processor = Pix2StructProcessor.from_pretrained("google/pix2struct-ai2d-large")107 108question = "What does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud"109 110inputs = processor(images=image, text=question, return_tensors="pt").to("cuda")111 112predictions = model.generate(**inputs)113print(processor.decode(predictions[0], skip_special_tokens=True))114>>> ash cloud115```116 117### In half precision, on GPU:118 119You can run the model in full precision on CPU:120```python121import requests122from PIL import Image123 124import torch125from transformers import Pix2StructForConditionalGeneration, Pix2StructProcessor126 127image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg"128image = Image.open(requests.get(image_url, stream=True).raw)129 130model = Pix2StructForConditionalGeneration.from_pretrained("google/pix2struct-ai2d-large", torch_dtype=torch.bfloat16).to("cuda")131processor = Pix2StructProcessor.from_pretrained("google/pix2struct-ai2d-large")132 133question = "What does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud"134 135inputs = processor(images=image, text=question, return_tensors="pt").to("cuda", torch.bfloat16)136 137predictions = model.generate(**inputs)138print(processor.decode(predictions[0], skip_special_tokens=True))139>>> ash cloud140```141 142 143# Contribution144 145This model was originally contributed by Kenton Lee, Mandar Joshi et al. and added to the Hugging Face ecosystem by [Younes Belkada](https://huggingface.co/ybelkada).146 147# Citation148 149If you want to cite this work, please consider citing the original paper:150```151@misc{https://doi.org/10.48550/arxiv.2210.03347,152  doi = {10.48550/ARXIV.2210.03347},153  154  url = {https://arxiv.org/abs/2210.03347},155  156  author = {Lee, Kenton and Joshi, Mandar and Turc, Iulia and Hu, Hexiang and Liu, Fangyu and Eisenschlos, Julian and Khandelwal, Urvashi and Shaw, Peter and Chang, Ming-Wei and Toutanova, Kristina},157  158  keywords = {Computation and Language (cs.CL), Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},159  160  title = {Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding},161  162  publisher = {arXiv},163  164  year = {2022},165  166  copyright = {Creative Commons Attribution 4.0 International}167}168```