pabloOmega/donut_hw_entities_v2

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. --> In this work, we study and train a model based on the Encoder-Decoder architecture (Donut) to generate structured text from handwritten text images. This approach extends beyond conventional Optical Character Recognition (OCR) by enabling the model to not only transcribe the text but also identify structural elements such as titles, subtitles, paragraphs, and equations. To facilitate training, a synthetic image generator was developed, capable of accurately simulating handwritten text using datasets such as IAM and CHROME. Furthermore, data augmentation techniques—including random rotation, Gaussian noise, and brightness and contrast adjustments—were employed to enhance the model’s generalization capabilities. As a result, the model outperformed alternatives such as Nougat in terms of Levenshtein distance, demonstrating improved transcription quality and structural understanding.

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

The most recent OCR models are based on complex architectures capable of generating text from images using an encoder-decoder framework, which leverages Transformers to process both image and text embeddings, thereby capturing intricate relationships \cite{Donut, layoutlmv3, nougat, swinVILT}. Embeddings are data vectors that typically represent words or tokens and can be processed by the Transformer, enhancing its ability to capture semantic relationships \cite{Pytorch2024-Embeddings}. However, to achieve the same with images, the input is segmented using a grid (patches), treating each cell as if it were a word. Each cell must then be flattened through Flatten layers to be treated as a one-dimensional vector \cite{vit}. This approach enables a more effective representation and learning process over images by exploiting the attention mechanisms within a Transformer \cite{Transformers}.

Despite the existence of OCR models with high accuracy, the transcription of handwritten text and the detection of entities such as titles, subtitles, formatting, handwritten equations, among others, remain a significant challenge for current models, including both commercial and open-source solutions \cite{TrOCR, googleOCR}. The motivation behind this study lies in the absence of a sufficiently reliable comprehensive solution that, through a single computer vision model equipped with the cutting-edge Transformer architecture \cite{Transformers}, is capable of recognizing the majority of entities within a handwritten document. Such a solution would greatly enhance and streamline the digitization of these documents. This study also aims to pave the way for future research to analyze the behavior of advanced computer vision models and develop novel solutions through Computing and Mathematics.

This work proposes the fine-tuning and evaluation of a multimodal image-to-text model based on the Transformer architecture, designed for the recognition and transcription of handwritten text into LaTeX format. The model should be capable of identifying document layout components, including titles, subtitles, paragraphs, tables, equations, and the textual content itself. Equations will also be processed and converted into LaTeX format. Given the clear need for data to train the model, the study considers the use of a synthetic image generator that simulates handwritten text with sufficient variability and generality, while also including entities such as equations and tables.

• —Developed by: Pablo Navarrete
• —Model type: Image-to-Text
• —Language(s) (NLP): English
• —License: Creative Commons Attribution 4.0 (you may use it for commercial purposes, but you must give credit to the author)
• —Finetuned from model: Donut Base

Model Sources

<!-- Provide the basic links for the model. -->

• —Repository: [More Information Needed]
• —Paper [optional]: [More Information Needed]
• —Demo [optional]: [More Information Needed]

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

This model is designed to generate structured text (e.g., paragraphs, titles, subtitles, and equations) from handwritten text images. It leverages the Donut architecture (Swin Transformer encoder + BART decoder) for document understanding and transcription. Its direct applications include:

• —Digitization of handwritten academic documents.
• —Conversion of handwritten notes to LaTeX-style structured text.
• —Extraction of semantic content from images of handwritten material for accessibility and storage.

Intended users are researchers and developers in the fields of computer vision and natural language processing, especially those interested in Optical Character Recognition (OCR) and document structure analysis.

Downstream Use

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

When fine-tuned or integrated into larger systems, the model can be used for:

• —Smart archival tools for handwritten historical or educational materials.
• —Educational apps that convert handwritten math/science content into digital form.
• —Assisting visually impaired users by converting handwritten notes into readable structured text via screen readers.

Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

This model is not suitable for:

• —Recognition of highly stylized handwriting or non-standard writing scripts not represented in the training data.
• —Security-critical applications (e.g., automated check processing, legal document signing).
• —Real-time OCR in resource-constrained edge devices due to its heavy model size and GPU dependence.

Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

The model is trained primarily on synthetically generated handwritten data and augmented samples. As such:

• —It may underperform with actual handwritten data that deviates significantly from training patterns (e.g., culturally diverse handwriting).
• —It may fail to generalize to low-light, noisy, or blurry images not represented in synthetic datasets.
• —Structural interpretation (e.g., correct identification of titles vs. subtitles) is prompt-dependent and not guaranteed to be perfect.

Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users should:

• —Avoid deploying the model in safety-critical or highly regulated settings without additional fine-tuning and validation.
• —Consider augmenting training with real-world handwritten datasets to improve generalization.
• —Post-process the model outputs to detect and correct potential structural or transcription errors.

How to Get Started with the Model

Use the code below to get started with the model. To get more accuracy, it's recommendable that you use images with 640x480 with a title, two subtitle and two paragraphs with equations.

from transformers import DonutProcessor, VisionEncoderDecoderModel
from PIL import Image
import requests
# load image from the IAM database
url = 'https://cdn-uploads.huggingface.co/production/uploads/65bc102dc1a44b6ef18be34b/65VBkKutGZ0WoERpfD8uw.jpeg'
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
processor = DonutProcessor.from_pretrained("pabloOmega/donut_hw_entities_v2")
model = VisionEncoderDecoderModel.from_pretrained("pabloOmega/donut_hw_entities_v2")
pixel_values = processor(image, return_tensors="pt").pixel_values

# To recognize the title use this prompt:
prompt = f"<s_hw><s_generate_title>"
decoder_input_ids = processor.tokenizer(prompt, add_special_tokens=False).input_ids
decoder_input_ids = torch.tensor(decoder_input_ids)

outputs = model.generate(pixel_values,
                            decoder_input_ids=decoder_input_ids.unsqueeze(0),
                            max_length=255,
                            pad_token_id=processor.tokenizer.pad_token_id,
                            eos_token_id=processor.tokenizer.convert_tokens_to_ids(f"</s_title>"),
                            use_cache=True,
                            num_beams=1,
                            bad_words_ids=[[processor.tokenizer.unk_token_id]],
                            return_dict_in_generate=True,
                            repetition_penalty=1.2,
                        )
predictions = []
for seq in processor.tokenizer.batch_decode(outputs.sequences):
    seq = seq.replace(processor.tokenizer.eos_token, "").replace(processor.tokenizer.pad_token, "")
    # seq = re.sub(r"<.*?>", "", seq, count=1).strip()  # remove first task start token
    predictions.append(seq)

predictions = predictions[0]
print(predictions)

# To recognize the subtitle use this prompt:
prompt = f"<s_hw><s_generate_subtitles>"
decoder_input_ids = processor.tokenizer(prompt, add_special_tokens=False).input_ids
decoder_input_ids = torch.tensor(decoder_input_ids)

outputs = model.generate(pixel_values,
                            decoder_input_ids=decoder_input_ids.unsqueeze(0),
                            max_length=255,
                            pad_token_id=processor.tokenizer.pad_token_id,
                            eos_token_id=processor.tokenizer.convert_tokens_to_ids(f"</s_subtitle>"),
                            use_cache=True,
                            num_beams=1,
                            bad_words_ids=[[processor.tokenizer.unk_token_id]],
                            return_dict_in_generate=True,
                            repetition_penalty=1.2,
                        )
predictions = []
for seq in processor.tokenizer.batch_decode(outputs.sequences):
    seq = seq.replace(processor.tokenizer.eos_token, "").replace(processor.tokenizer.pad_token, "")
    # seq = re.sub(r"<.*?>", "", seq, count=1).strip()  # remove first task start token
    predictions.append(seq)

predictions = predictions[0]
print(predictions)

# To recognize the paragraph use this prompt:
prompt = f"<s_hw><s_generate_paragraphs>"
decoder_input_ids = processor.tokenizer(prompt, add_special_tokens=False).input_ids
decoder_input_ids = torch.tensor(decoder_input_ids)

outputs = model.generate(pixel_values,
                            decoder_input_ids=decoder_input_ids.unsqueeze(0),
                            max_length=255,
                            pad_token_id=processor.tokenizer.pad_token_id,
                            eos_token_id=processor.tokenizer.convert_tokens_to_ids(f"</s_paragraph>"),
                            use_cache=True,
                            num_beams=1,
                            bad_words_ids=[[processor.tokenizer.unk_token_id]],
                            return_dict_in_generate=True,
                            repetition_penalty=1.2,
                        )
predictions = []
for seq in processor.tokenizer.batch_decode(outputs.sequences):
    seq = seq.replace(processor.tokenizer.eos_token, "").replace(processor.tokenizer.pad_token, "")
    # seq = re.sub(r"<.*?>", "", seq, count=1).strip()  # remove first task start token
    predictions.append(seq)

predictions = predictions[0]
print(predictions)

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

The training data consists of synthetically generated handwritten images simulating paragraphs, titles, subtitles, and equations. These were constructed using:

• —IAM dataset (handwriting stroke coordinates).
• —Equation generator from CHROME 2014 Dataset.
• —Custom generators for layout variation.

You could find the training dataset here.

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. --> If you want to fine-tuning the model, you may want to review this notebook.

Preprocessing

Images were preprocessed with padding and normalization. Text was tokenized into structured prompts representing document elements. You just need to load the processor.

Training Hyperparameters

• —Learning rate: 3e-5
• —Batch size: 4
• —Optimizer: AdamW
• —Epochs: 14
• —Hardware: Google Colab VM with NVIDIA A100 GPU, 90GB CPU RAM.

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data, Factors & Metrics

Testing Data

<!-- This should link to a Dataset Card if possible. -->

• —Synthetic handwritten documents.
• —Real handwritten images from modified IAM dataset.

Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

Evaluation differentiated between models capable of processing printed vs. handwritten content, and the ability to recognize structured elements.

Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

• —BLEU Score: Measures overlap between generated and reference tokens.
• —Levenshtein Distance: Measures edit distance between generated and true sequences.

Results

Technical Specifications

Model Architecture and Objective

• —Encoder: Swin Transformer
• —Decoder: BART
• —Objective: Structured text generation from handwritten images using OCR and prompt-based decoding.

Citation

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

BibTeX:

@mastersthesis{navarrete2025donutocr, author = {Pablo Steve Navarrete Arroyo}, title = {Structured Text Generation from Handwritten Documents using Donut Architecture}, school = {Universidad Internacional de la Rioja}, year = {2025} }

APA:

Navarrete Arroyo, P. S. (2025). Structured Text Generation from Handwritten Documents using Donut Architecture (Master’s thesis). Máster en Ingeniería Matemática y Computación. Universidad Internacional de la Rioja.

Model Card Contact

E-mail