usmanadaudu/lightonocr-blindspots

LightOnOCR Blind Spots Datasets

Dataset Summary

This dataset contains examples where the OCR model LightOnOCR-2-1B-base produces incorrect predictions.

The dataset was created to analyze failure cases and identify blind spots in the model.

Each dataset entry contains:

• —image_url - URL of the image
• —expected_output - Correct transcription of the image
• —model_output - Text generated by the model

The goal of this dataset is to help diagnose model weaknesses and guide future fine-tuning efforts.

Model Evaluation

Model tested:

https://huggingface.co/lightonai/LightOnOCR-2-1B-base

LightOnOCR-2-1B-base is a vision-language model designed for OCR tasks such as text extraction from document images.

How Model Was Loaded

The model was run in Google Colab using the Huggingface Transformers library.

Model Setup

The model setup was done using the code below

import torch
from transformers import LightOnOcrForConditionalGeneration, LightOnOcrProcessor

device = "cuda" if torch.cuda.is_available() else "cpu"

model = LightOnOcrForConditionalGeneration.from_pretrained(
    "lightonai/LightOnOCR-2-1B-base"
).to(device)

processor = LightOnOcrProcessor.from_pretrained(
    "lightonai/LightOnOCR-2-1B-base"
) 

Inference

Outputs were gotten from the model using the code below

url = "link/to/image"

conversation = [
    {"role": "user",
     "content": [{"type": "image", "url": url}]}
]

inputs = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt"
)

inputs = {k: v.to(device) for k, v in inputs.items()}

output_ids = model.generate(**inputs, max_new_tokens=512)

generated_ids = output_ids[0, inputs["input_ids"].shape[1]:]

text = processor.decode(generated_ids, skip_special_tokens=True)

print(text)

Images were passed to the model and the generated OCR output was recorded.

Dataset Creation

Images used in this dataset include several challenging OCR handwritten notes cases which includes:

• —tables
• —mathematical formulas
• —low-light images
• —documents on rotated planes

For each image:

1. 1.The correct text transcription was manually recorded.
2. 2.The model was run on the image.
3. 3.The model output was compared to the expected output.

Only examples where the model produced incorrect or degraded predictions were included.

Observed Model Blind Spots

During testing, the model showed weaknesses in the following cases:

Handwritten Text

Handwritten characters are frequently misrecognized.

Table Structure

The model gives incorrect predictions mostly in cases where table cells spans multiple rows or columns. Likewise when a column contains sequence of numbers, the model tends to only print out number sequnce far beyond what is in the image

Mathematical Notation

Symbols such as superscripts and sunbscripts are usually incorrectly parsed.

Low-Quality Images

Blurred, low-light or noisy images significantly reduce recognition accuracy.

Potential Improvements

The model could be improved by fine-tuning on datasets of handwritten documents containing:

• —structured tables especially ones having cells that span multiple rows or columns
• —scientific equations especially those having superscripts and subscripts
• —complex document layouts

Relevant datasets include:

• —DocLayNet
• —PubLayNet
• —SROIE

Estimated Dataset Size Needed

To significantly improve robustness, a fine-tuning dataset of approximately 50,000 -- 200,000 annotated document images would likely be required.

The dataset should include handwritten documents with a diverse mixture of:

• —receipts
• —tables
• —scientific papers
• —degraded images

Limitations

This dataset only contains failure cases and does not represent the full performance of the model. It is intended for diagnostic analysis rather than benchmarking.

License

MIT License