erayalp/blip2-flan-t5-xl-LoRA-image-captioning

Model Summary

This model is a fine-tuned version of BLIP-2 with the flan-t5-xl language decoder, optimized for image captioning tasks. Fine-tuning was performed using LoRA (Low-Rank Adaptation) for parameter-efficient adaptation. The training objective was to generate high-quality, semantically rich captions for images from the Open Images dataset.

It was developed for a captioning competition evaluated using Fréchet GTE Distance (FGD), which uses GTE-small embeddings to assess the alignment of image and caption semantics.

Training Objective

• —Task: Image Captioning
• —Base Model: Salesforce/blip2-flan-t5-xl
• —Backbone: Frozen ViT-G + frozen Q-Former
• —Decoder: Fine-tuned flan-t5-xl with LoRA
• —Loss: Cross-entropy with optional GTE-aware auxiliary loss
• —Evaluation: Fréchet GTE Distance (FGD) between image and caption embeddings

⸻

Dataset

• —Training Dataset: Subset of Open Images with curated image-caption pairs
• —Augmentation: Synthetic captions
• —Image Features: Preprocessed using BLIP-2’s frozen vision encoder

⸻

Fine-Tuning Configuration

• —LoRA Rank: 128
• —Alpha: 128
• —Dropout: 0.05
• —Target Modules q, v, and k in attention blocks
• —Precision: bfloat16
• —Optimizer: AdamW
• —Learning Rate: 3e-5
• —Scheduler: Cosine with warmup
• —Batch Size: 32
• —Epochs: 4
• —Accumulation: 2 gradient accumulation steps
• —Logging: Weights & Biases

⸻

Performance

• —Fréchet GTE Distance (↓), Achieved competitive score in the competition
• —Caption Quality: High semantic alignment and fluency
• —Improved OCR capabilities

⸻

Usage

from transformers import BlipProcessor, BlipForConditionalGeneration
from PIL import Image
import torch

processor = BlipProcessor.from_pretrained("erayalp/blip2-flan-t5-xl-LoRA-image-captioning")
model = BlipForConditionalGeneration.from_pretrained("erayalp/blip2-flan-t5-xl-LoRA-image-captioning").to("cuda")

img = Image.open("example.jpg").convert("RGB")
prompt = "Provide a detailed caption for this photo."
prompts = [prompt] * len(images)
inputs = processor(images=img, text=prompts, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=30)
caption = processor.decode(outputs[0], skip_special_tokens=True)

print(caption)

⸻

Limitations

• —May underperform on out-of-domain images or very abstract concepts
• —Quality of captions may vary depending on scene complexity
• —Not trained on video or temporal sequences