CoolFace
Modelpublic

ericmrib/math-ocr

sourceHugging Faceupdated 4mo agoView on Hugging Face
0likes4downloads
Model Card

Model Card for Model ID

This is a LoRA adapter fine-tuned on the Qwen2.5-VL-3B-Instruct base model for Optical Character Recognition (OCR) of mathematical formulas, converting images directly into LaTeX code.

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

This model is a Vision-Language Model (VLM) adapter fine-tuned to transcribe images of math equations into their corresponding LaTeX representation. It uses 4-bit QLoRA quantization to achieve efficient fine-tuning on a single consumer-grade GPU (e.g., Tesla T4).

  • —Developed by: ericmrib
  • —Model type: Vision-Language Model Adapter (LoRA)
  • —Language(s) (NLP): English, LaTeX
  • —Finetuned from model: Qwen/Qwen2.5-VL-3B-Instruct

Out-of-Scope Use

The model is highly specialized for LaTeX formula extraction and may not perform well on general image captioning, document parsing without equations, or natural language reasoning tasks outside its fine-tuned scope.

Bias, Risks, and Limitations

As with any vision-language model, the performance relies heavily on the quality, clarity, and handwriting style (if applicable) of the input images. Complex or highly stylized formatting might lead to inaccuracies in the generated LaTeX.

How to Get Started with the Model

Use the code below to get started with the model.

python
from peft import PeftModel
from transformers import AutoProcessor, AutoModelForImageTextToText

# Load base model and processor
base_model = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct", device_map="auto")
processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct")

# Load adapter
model = PeftModel.from_pretrained(base_model, "ericmrib/math-ocr")

Training Details

Training Data

The model was fine-tuned on the unsloth/LaTeX_OCR dataset, which contains approximately 76,000 samples of image-to-LaTeX pairs.

Training Hyperparameters
  • —Training regime: fp16 mixed precision
  • —Quantization: 4-bit (QLoRA) with NF4
  • —Batch size: 1 (with 16 gradient accumulation steps; effective batch size = 16)
  • —Learning rate: 2e-4
  • —Optimizer: adamw_torch
  • —Scheduler: cosine
  • —Max Steps: 500
  • —LoRA configuration: r=16, alpha=32, dropout=0.05, targetmodules=["qproj", "kproj", "vproj", "oproj", "gateproj", "upproj", "downproj"]

Technical Specifications

Hardware
  • —Hardware Type: Tesla T4 (1x)
  • —Hours used: ~4.2 hours
  • —Cloud Provider: Google Cloud (Google Colab)