CoolFace
Modelpublic

phucdu/qwen2.5-vl-3b-mjsynth-ocr-lora

sourceHugging Faceupdated 2mo agoView on Hugging Face
2likes7downloads
Model Card

Model Card for phucdu/qwen2.5-vl-3b-mjsynth-ocr-lora

This model is a LoRA fine-tuned version of Qwen/Qwen2.5-VL-3B-Instruct specifically optimized for Optical Character Recognition (OCR) tasks using the MJSynth dataset.

Model Details

Model Description

This model utilizes Parameter-Efficient Fine-Tuning (PEFT) via LoRA and 4-bit quantization to adapt the base Qwen2.5-VL-3B-Instruct model for highly accurate text recognition from images. It achieves significant improvements in Character Error Rate (CER) and Exact Match accuracy compared to the base model.

  • —Developed by: phucdu
  • —Model type: Vision-Language Model (VLM) / LoRA Adapter
  • —Language(s) (NLP): English
  • —License: Apache 2.0 (inheriting from Qwen)
  • —Finetuned from model: Qwen/Qwen2.5-VL-3B-Instruct

Uses

Direct Use

Extracting and reading text from images (Optical Character Recognition - OCR). It is particularly tuned on single-word crops but generalizes well to standard text reading tasks.

How to Get Started with the Model

(Mục này đã được để trống theo yêu cầu của tác giả - Không chứa code tải mô hình).

Training Details

Training Data

A subset of priyank-m/MJSynth_text_recognition.

  • —Pool size: 8000 examples
  • —Split: 6400 Train / 800 Validation / 800 Test.

Training Procedure

Training Hyperparameters
  • —Training regime: LoRA with 4-bit quantization (nf4, double quant) and bfloat16 compute dtype.
  • —LoRA Rank (r): 16
  • —LoRA Alpha: 32
  • —LoRA Dropout: 0.05
  • —Target Modules: down_proj, gate_proj, k_proj, o_proj, q_proj, up_proj, v_proj
  • —Epochs: 3
  • —Batch Size: 16 (Train) / 8 (Eval)
  • —Learning Rate: 1e-4
  • —LR Scheduler: Cosine
  • —Warmup Ratio: 0.03
  • —Weight Decay: 0.0
Speeds, Sizes, Times
  • —Hardware: 1x NVIDIA L4 (24GB) GPU on Google Colab Pro.
  • —Training Time: ~35 minutes (2115 seconds for 3 epochs).

Evaluation

Testing Data, Factors & Metrics

  • —Testing Data: 800 held-out samples from the MJSynth dataset.
  • —Metrics: Character Error Rate (CER), Word Error Rate (WER), Exact Match Accuracy.

Results

Comparing Base Model vs. Fine-tuned Model on the test set:

  • —CER: 1.3056 ➔ 0.0842 (93.5% relative improvement)
  • —WER: 2.3462 ➔ 0.1437 (93.8% relative improvement)
  • —Exact Match Accuracy: 62.37% ➔ 85.62% (37.2% relative improvement)
  • —Avg Inference Latency: ~0.768 sec/image
Summary

The LoRA fine-tuning process significantly enhanced the model's OCR capabilities on the target dataset, bringing the exact match accuracy up to over 85% while massively dropping both CER and WER.