TomasFAV/Pix2StructCzechInvoiceV0123

Pix2StructCzechInvoice (V3 – Full Pipeline with Real Data Fine-Tuning)

This model is a fine-tuned version of google/pix2struct-docvqa-base for structured information extraction from Czech invoices.

It achieves the following results on the evaluation set:

• —Loss: 0.1542
• —F1: 0.8404

Model description

Pix2StructCzechInvoice (V3) is the final generative model in the experimental pipeline.

Unlike token classification approaches, this model:

• —processes full document images
• —generates structured outputs as text sequences

It extracts key invoice fields such as:

• —supplier
• —customer
• —invoice number
• —bank details
• —totals
• —dates

By combining synthetic, hybrid, and real data, this version significantly improves both performance and stability.

Training data

The dataset used in this stage combines:

1. 1.Synthetic template-based invoices (V0)
2. 2.Synthetic invoices with randomized layouts (V1)
3. 3.Hybrid invoices with real layouts and synthetic content (V2)
4. 4.Real annotated invoices

Real data fine-tuning

The final stage introduces:

• —real invoice images
• —realistic visual noise and distortions
• —natural language variability
• —real formatting inconsistencies

This allows the model to:

• —better align generated outputs with real-world distributions
• —improve robustness of sequence generation
• —reduce hallucinations and formatting errors

Role in the pipeline

This model corresponds to:

V3 – Full pipeline (synthetic + hybrid + real data fine-tuning)

It represents:

• —the final generative model
• —the best-performing Pix2Struct variant
• —an end-to-end extraction approach

Intended uses

• —End-to-end invoice information extraction from images
• —Document VQA and generative document understanding
• —OCR-free document processing pipelines
• —Research in generative vs structured extraction approaches

Limitations

• —Output format may still be inconsistent
• —Sensitive to decoding strategy and prompt structure
• —Less interpretable than token classification models
• —Requires post-processing for structured outputs
• —Computationally more expensive

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• —learning_rate: 0.0001
• —trainbatchsize: 8
• —evalbatchsize: 1
• —seed: 42
• —optimizer: Use OptimizerNames.ADAMWTORCHFUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• —lrschedulertype: cosinewithrestarts
• —lrschedulerwarmup_steps: 0.1
• —num_epochs: 10
• —mixedprecisiontraining: Native AMP

Training results

Framework versions

• —Transformers 5.0.0
• —PyTorch 2.10.0+cu128
• —Datasets 4.0.0
• —Tokenizers 0.22.2