TANZEELDATA/blip-vqa-ocr-app

BLIP VQA — Fine-tuned on OCR-VQA

This model is `Salesforce/blip-vqa-base` fine-tuned on the `howard-hou/OCR-VQA` dataset, which contains book cover images with question-answer pairs focused on reading text visible in images (OCR-style VQA).

Model Details

Training Objective

The model was fine-tuned in generative mode: the answer decoder is supervised with cross-entropy on the tokenised answer string, making it suitable for open-ended, multi-word answers via model.generate().

⚠️ Note on size-mismatch warning You may see a warning about text_decoder.cls.predictions.* weights when loading the checkpoint. This is expected and harmless — those classification head weights were part of the original pre-trained model but are not used during generative inference. All answers are produced via generate().

Quick Start

from transformers import BlipProcessor, BlipForQuestionAnswering
from PIL import Image
import requests, torch

repo_id   = "YOUR_USERNAME/blip-vqa-ocr-finetuned"   # ← update
processor = BlipProcessor.from_pretrained(repo_id)
model     = BlipForQuestionAnswering.from_pretrained(repo_id)
model.eval()

# Load any book-cover image
url   = "https://covers.openlibrary.org/b/id/8739161-L.jpg"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")

question = "What is the title of this book?"

inputs = processor(images=image, text=question, return_tensors="pt")
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=20, num_beams=4)

print(processor.decode(out[0], skip_special_tokens=True))

Evaluation (200 validation samples)

Architecture

Image ──────► ViT Encoder  ─┐
                             ├─► Cross-Attention Fusion ─► Answer Decoder ─► Answer Text
Question ──► BERT Encoder ──┘

BLIP (Bootstrapping Language-Image Pre-training) uses:

• —A Vision Transformer (ViT) as the image encoder
• —A BERT-based text encoder / decoder with cross-attention fusion
• —End-to-end training with multimodal objectives

Limitations

• —Fine-tuned on a small subset (3 k samples) — larger training runs will improve accuracy significantly.
• —Best suited for OCR-style VQA: book covers, documents, signs, labels.
• —May hallucinate on out-of-distribution images or complex visual scenes.
• —Short answers (1–5 words) are most reliable.

Dataset

`howard-hou/OCR-VQA` contains ~200 k book cover images sourced from Google Books, each with multiple question-answer pairs asking about text visible in the cover (title, author, publisher, year, genre, etc.).

Citation

@misc{li2022blip,
  title         = {BLIP: Bootstrapping Language-Image Pre-training for
                   Unified Vision-Language Understanding and Generation},
  author        = {Junnan Li and Dongxu Li and Caiming Xiong and Steven Hoi},
  year          = {2022},
  eprint        = {2201.12086},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV}
}