lamao-ab/paligemma-blind-assist-lora-merged-v1

👁️ PaliGemma-BlindAssist: A parameter-efficient vision-language model adapter for blind assistance VQA & Captioning using LoRA

<div align="center">

   

</div>

🎯 Model Overview

A full-precision (bfloat16) adaptation of `google/paligemma-3b-mix-224` for accessibility vision-language tasks, trained with LoRA on the VizWiz datasets and merged into a standalone checkpoint. It performs visual question answering and scene captioning for blind and low-vision users, and represents the quality ceiling of our adaptation: the 4-bit QLoRA variant matches it to within a fraction of a point at 40% of the memory footprint, and is the variant intended for on-device deployment.

Adaptation improves the target task substantially while leaving general-domain performance largely intact. On VizWiz captioning, CIDEr-D rises from 55.31 to 98.08 and mean caption length from 5.03 to 10.30 words, matching the dataset's more descriptive reference style; VizWiz-VQA accuracy rises from 73.95 to 75.80. General-domain performance declines only modestly, by 0.50 points on VQAv2 and 5% relative on COCO-Caps.

🔧 Specifications

• —Base: google/paligemma-3b-mix-224 (bfloat16)
• —Adapter: LoRA, rank 8, alpha 16, dropout 0.05 · 11.3 M trainable params (0.385 %)
• —Target modules: q/k/v/o_proj, gate/up/down_proj
• —Data: 137,678 train / 27,569 val · seed 123
• —Schedule: 3 epochs, 3,225 steps · effective batch 128 (16 × 8 grad-accum) · max seq 512
• —Optimiser: adamw_torch_fused · LR 2e-4, cosine (warmup 0.03)
• —Parameters: 2.92 B (adapter merged into the base weights)
• —Precision: bf16 (merged, standalone)
• —In-memory size: 5.45 GB (model.get_memory_footprint())
• —Training peak VRAM: 53.1 GB

📊 Measured Performance

Blind-domain (target task) — VizWiz

VizWiz-VQA | Model | Overall | Yes/No | Number | Other | Unans. | |-------|:-------:|:------:|:------:|:-----:|:------:| | Base | 73.95 | 88.71 | 66.10 | 66.10 | 91.67 | | LoRA | 75.80 ±0.21 | 86.55 | 67.56 | 66.54 | 97.73 |

VizWiz-Caps | Model | CIDEr-D | BLEU-4 | METEOR | ROUGE-L | Len. | |-------|:-------:|:------:|:------:|:-------:|:----:| | Base | 55.31 | 12.07 | 14.08 | 28.76 | 5.03 | | LoRA | 98.08 ±1.51 | 30.77 ±0.36 | 23.45 ±0.31 | 49.54 ±0.47 | 10.30 |

General-domain (control) — VQAv2 / COCO-Caps

VQAv2 | Model | Overall | Yes/No | Number | Other | |-------|:-------:|:------:|:------:|:-----:| | Base | 81.65 | 94.74 | 67.53 | 73.49 | | LoRA | 81.15 ±0.04 | 94.57 | 66.38 | 72.85 |

COCO-Caps | Model | CIDEr-D | BLEU-4 | METEOR | ROUGE-L | Len. | |-------|:-------:|:------:|:------:|:-------:|:----:| | Base | 131.21 | 31.96 | 30.62 | 59.17 | 12.40 | | LoRA | 124.54 ±1.36 | 34.58 ±0.39 | 30.63 ±0.06 | 58.62 ±0.19 | 11.20 |

_{LoRA/QLoRA scores are mean ± std over 3 seeds (42, 123, 7). This checkpoint is the seed-123 run.}

🚀 Usage

from transformers import PaliGemmaForConditionalGeneration, PaliGemmaProcessor
from PIL import Image
import torch

repo = "lamao-ab/paligemma-blind-assist-lora-merged-v1"
model = PaliGemmaForConditionalGeneration.from_pretrained(
    repo, torch_dtype=torch.bfloat16, device_map="auto"
)
processor = PaliGemmaProcessor.from_pretrained(repo)

image = Image.open("example.jpg").convert("RGB")
prompt = "describe the scene for a blind person"  
inputs = processor(text=prompt, images=image, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128)
print(processor.decode(out[0], skip_special_tokens=True))

📄 License

Inherits the Gemma license from the base model.