philschmid/layoutlm-funsd

1---2tags:3- generated_from_trainer4- endpoints-template5library_name: generic6datasets:7- funsd8model-index:9- name: layoutlm-funsd10  results: []11pipeline_tag: other12---13 14<!-- This model card has been generated automatically according to the information the Trainer had access to. You15should probably proofread and complete it, then remove this comment. -->16 17# layoutlm-funsd18 19This model is a fine-tuned version of [microsoft/layoutlm-base-uncased](https://huggingface.co/microsoft/layoutlm-base-uncased) on the funsd dataset.20It achieves the following results on the evaluation set:21- Loss: 1.004522- Answer: {'precision': 0.7348314606741573, 'recall': 0.8084054388133498, 'f1': 0.7698646262507357, 'number': 809}23- Header: {'precision': 0.44285714285714284, 'recall': 0.5210084033613446, 'f1': 0.47876447876447875, 'number': 119}24- Question: {'precision': 0.8211009174311926, 'recall': 0.8403755868544601, 'f1': 0.8306264501160092, 'number': 1065}25- Overall Precision: 0.759926- Overall Recall: 0.808327- Overall F1: 0.786628- Overall Accuracy: 0.810629 30### Training hyperparameters31 32The following hyperparameters were used during training:33- learning_rate: 3e-0534- train_batch_size: 1635- eval_batch_size: 836- seed: 4237- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-0838- lr_scheduler_type: linear39- num_epochs: 1540- mixed_precision_training: Native AMP41 42## Deploy Model with Inference Endpoints43 44Before we can get started, make sure you meet all of the following requirements:45 461. An Organization/User with an active plan and *WRITE* access to the model repository.472. Can access the UI: [https://ui.endpoints.huggingface.co](https://ui.endpoints.huggingface.co/endpoints)48 49 50 51### 1. Deploy LayoutLM and Send requests52 53In this tutorial, you will learn how to deploy a [LayoutLM](https://huggingface.co/docs/transformers/model_doc/layoutlm) to [Hugging Face Inference Endpoints](https://huggingface.co/inference-endpoints) and how you can integrate it via an API into your products. 54 55This tutorial is not covering how you create the custom handler for inference. If you want to learn how to create a custom Handler for Inference Endpoints, you can either checkout the [documentation](https://huggingface.co/docs/inference-endpoints/guides/custom_handler) or go through [“Custom Inference with Hugging Face Inference Endpoints”](https://www.philschmid.de/custom-inference-handler) 56 57We are going to deploy [philschmid/layoutlm-funsd](https://huggingface.co/philschmid/layoutlm-funsd) which implements the following `handler.py` 58 59```python60from typing import Dict, List, Any61from transformers import LayoutLMForTokenClassification, LayoutLMv2Processor62import torch63from subprocess import run64 65# install tesseract-ocr and pytesseract66run("apt install -y tesseract-ocr", shell=True, check=True)67run("pip install pytesseract", shell=True, check=True)68 69# helper function to unnormalize bboxes for drawing onto the image70def unnormalize_box(bbox, width, height):71    return [72        width * (bbox[0] / 1000),73        height * (bbox[1] / 1000),74        width * (bbox[2] / 1000),75        height * (bbox[3] / 1000),76    ]77 78# set device79device = torch.device("cuda" if torch.cuda.is_available() else "cpu")80 81class EndpointHandler:82    def __init__(self, path=""):83        # load model and processor from path84        self.model = LayoutLMForTokenClassification.from_pretrained(path).to(device)85        self.processor = LayoutLMv2Processor.from_pretrained(path)86 87    def __call__(self, data: Dict[str, bytes]) -> Dict[str, List[Any]]:88        """89        Args:90            data (:obj:):91                includes the deserialized image file as PIL.Image92        """93        # process input94        image = data.pop("inputs", data)95 96        # process image97        encoding = self.processor(image, return_tensors="pt")98 99        # run prediction100        with torch.inference_mode():101            outputs = self.model(102                input_ids=encoding.input_ids.to(device),103                bbox=encoding.bbox.to(device),104                attention_mask=encoding.attention_mask.to(device),105                token_type_ids=encoding.token_type_ids.to(device),106            )107            predictions = outputs.logits.softmax(-1)108 109        # post process output110        result = []111        for item, inp_ids, bbox in zip(112            predictions.squeeze(0).cpu(), encoding.input_ids.squeeze(0).cpu(), encoding.bbox.squeeze(0).cpu()113        ):114            label = self.model.config.id2label[int(item.argmax().cpu())]115            if label == "O":116                continue117            score = item.max().item()118            text = self.processor.tokenizer.decode(inp_ids)119            bbox = unnormalize_box(bbox.tolist(), image.width, image.height)120            result.append({"label": label, "score": score, "text": text, "bbox": bbox})121        return {"predictions": result}122```123 124### 2. Send HTTP request using Python125 126Hugging Face Inference endpoints can directly work with binary data, this means that we can directly send our image from our document to the endpoint. We are going to use `requests` to send our requests. (make your you have it installed `pip install requests`)127 128```python129import json130import requests as r131import mimetypes132 133ENDPOINT_URL="" # url of your endpoint134HF_TOKEN="" # organization token where you deployed your endpoint135 136def predict(path_to_image:str=None):137    with open(path_to_image, "rb") as i:138      b = i.read()139    headers= {140        "Authorization": f"Bearer {HF_TOKEN}",141        "Content-Type": mimetypes.guess_type(path_to_image)[0]142    }143    response = r.post(ENDPOINT_URL, headers=headers, data=b)144    return response.json()145 146prediction = predict(path_to_image="path_to_your_image.png")147 148print(prediction)149# {'predictions': [{'label': 'I-ANSWER', 'score': 0.4823932945728302, 'text': '[CLS]', 'bbox': [0.0, 0.0, 0.0, 0.0]}, {'label': 'B-HEADER', 'score': 0.992474377155304, 'text': 'your', 'bbox': [1712.529, 181.203, 1859.949, 228.88799999999998]},150```151 152 153### 3. Draw result on image154 155To get a better understanding of what the model predicted you can also draw the predictions on the provided image. 156 157```python158from PIL import Image, ImageDraw, ImageFont159 160# draw results on image161def draw_result(path_to_image,result):162  image = Image.open(path_to_image)163  label2color = {164      "B-HEADER": "blue",165      "B-QUESTION": "red",166      "B-ANSWER": "green",167      "I-HEADER": "blue",168      "I-QUESTION": "red",169      "I-ANSWER": "green",170  }171 172  # draw predictions over the image173  draw = ImageDraw.Draw(image)174  font = ImageFont.load_default()175  for res in result:176      draw.rectangle(res["bbox"], outline="black")177      draw.rectangle(res["bbox"], outline=label2color[res["label"]])178      draw.text((res["bbox"][0] + 10, res["bbox"][1] - 10), text=res["label"], fill=label2color[res["label"]], font=font)179  return image180 181draw_result("path_to_your_image.png", prediction["predictions"])182```