Norm/ERNIE-Layout-Pytorch

1---2license: mit3---4 5# ERNIE-Layout_Pytorch6 7- **Model type:** [ERNIE-Layout](https://arxiv.org/abs/2210.06155)8- **Repository:** [source code](https://github.com/NormXU/ERNIE-Layout-Pytorch): an unofficial ERNIE-Layout implementation in Pytorch9  10- **Converted from:** [PaddlePaddle/ernie-layoutx-base-uncased](https://huggingface.co/PaddlePaddle/ernie-layoutx-base-uncased)11  12 13The ERNIE-Layout-Pytorch model is initially released by [PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP). To make Pytorch users easy to use, the model has been converted into PyTorch format with the [tools/convert2torch.py](https://github.com/NormXU/ERNIE-Layout-Pytorch/blob/main/tools/convert2torch.py) script.14Please feel free to make any changes you need. For more details and use cases, please check the repo.15 16**A Quick Example**17```python18import torch19from PIL import Image20import torch.nn.functional as F21from networks import ErnieLayoutConfig, ErnieLayoutForQuestionAnswering, \22    ErnieLayoutProcessor, ErnieLayoutTokenizerFast23from transformers.models.layoutlmv3 import LayoutLMv3ImageProcessor24 25pretrain_torch_model_or_path = "Norm/ERNIE-Layout-Pytorch"26doc_imag_path = "./dummy_input.jpeg"27 28context = ['This is an example sequence', 'All ocr boxes are inserted into this list']29layout = [[381, 91, 505, 115], [738, 96, 804, 122]]  # make sure  all boxes are normalized between 0 - 100030pil_image = Image.open(doc_imag_path).convert("RGB")31 32# initialize tokenizer33tokenizer = ErnieLayoutTokenizerFast.from_pretrained(pretrained_model_name_or_path=pretrain_torch_model_or_path)34 35# initialize feature extractor36feature_extractor = LayoutLMv3ImageProcessor(apply_ocr=False)37processor = ErnieLayoutProcessor(image_processor=feature_extractor, tokenizer=tokenizer)38 39# Tokenize context & questions40question = "what is it?"41encoding = processor(pil_image, question, context, boxes=layout, return_tensors="pt")42 43# dummy answer start && end index44start_positions = torch.tensor([6])45end_positions = torch.tensor([12])46 47# initialize config48config = ErnieLayoutConfig.from_pretrained(pretrained_model_name_or_path=pretrain_torch_model_or_path)49config.num_classes = 2  # start and end50 51# initialize ERNIE for VQA52model = ErnieLayoutForQuestionAnswering.from_pretrained(53    pretrained_model_name_or_path=pretrain_torch_model_or_path,54    config=config,55)56 57output = model(**encoding, start_positions=start_positions, end_positions=end_positions)58 59# decode output60start_max = torch.argmax(F.softmax(output.start_logits, dim=-1))61end_max = torch.argmax(F.softmax(output.end_logits, dim=-1)) + 1  # add one ##because of python list indexing62answer = tokenizer.decode(encoding.input_ids[0][start_max: end_max])63print(answer)64 65 66```