lamao-ab/paligemma-blind-assist-qlora-merged-v1

👁️ PaliGemma-BlindAssist: A parameter-efficient vision-language model adapter for blind assistance VQA & Captioning using QLoRA

<div align="center">

   

</div>

🎯 Model Overview

A 4-bit (NF4) quantized adaptation of `google/paligemma-3b-mix-224` for accessibility vision-language tasks, trained with QLoRA on the VizWiz datasets and merged into a standalone checkpoint. It performs visual question answering and scene captioning for blind and low-vision users, and is intended for on-device inference on memory-constrained hardware such as the NVIDIA Jetson Orin Nano.

Quantization is close to free in task terms. Against the full-precision LoRA model trained under identical settings, this variant scores 75.71 vs 75.80 on VizWiz-VQA and 97.44 vs 98.08 CIDEr-D on VizWiz-Caps — differences of 0.09 and 0.64 points — while reducing the deployed footprint from 5.45 GB to 2.13 GB.

🔧 Specifications

• —Base: google/paligemma-3b-mix-224 (4-bit NF4 quantised)
• —Adapter: rank 8, alpha 16, dropout 0.05 · 11.3 M trainable params (0.385 %)
• —Target modules: q/k/v/o_proj, gate/up/down_proj
• —Data: 137,678 train / 27,569 val · seed 123
• —Schedule: 3 epochs, 3,225 steps · effective batch 128 (16 × 8 grad-accum) · max seq 512
• —Optimiser: paged_adamw_8bit · LR 2e-4, cosine (warmup 0.03)
• —Parameters: 2.92 B (adapter merged into the base weights)
• —Precision: 4-bit NF4, double-quantized, bfloat16 compute
• —In-memory size: 2.13 GB (model.get_memory_footprint())
• —Training peak VRAM: 64.4 GB

📊 Measured Performance

Blind-domain (target task) — VizWiz

VizWiz-VQA | Model | Overall | Yes/No | Number | Other | Unans. | |-------|:-------:|:------:|:------:|:-----:|:------:| | Base | 73.95 | 88.71 | 66.10 | 66.10 | 91.67 | | QLoRA | 75.71 ±0.14 | 86.54 | 63.88 | 66.51 | 97.68 |

VizWiz-Caps | Model | CIDEr-D | BLEU-4 | METEOR | ROUGE-L | Len. | |-------|:-------:|:------:|:------:|:-------:|:----:| | Base | 55.31 | 12.07 | 14.08 | 28.76 | 5.03 | | QLoRA | 97.44 ±1.45 | 30.63 ±0.29 | 23.30 ±0.35 | 49.33 ±0.38 | 10.24 |

General-domain (control) — VQAv2 / COCO-Caps

VQAv2 | Model | Overall | Yes/No | Number | Other | |-------|:-------:|:------:|:------:|:-----:| | Base | 81.65 | 94.74 | 67.53 | 73.49 | | QLoRA | 80.72 ±0.04 | 94.38 | 65.20 | 72.39 |

COCO-Caps | Model | CIDEr-D | BLEU-4 | METEOR | ROUGE-L | Len. | |-------|:-------:|:------:|:------:|:-------:|:----:| | Base | 131.21 | 31.96 | 30.62 | 59.17 | 12.40 | | QLoRA | 123.34 ±1.75 | 34.26 ±0.60 | 30.44 ±0.09 | 58.39 ±0.28 | 11.22 |

_{LoRA/QLoRA scores are mean ± std over 3 seeds (42, 123, 7). This checkpoint is the seed-123 run.}

On-device characterization (Jetson Orin Nano 8 GB, 15 W)

Scope. These figures characterize model-level execution on a development board. The model has not been evaluated with blind or visually impaired users, and is not a validated assistive product.

🚀 Usage

# 4-bit checkpoint — quantization config travels with the model, no BitsAndBytesConfig needed.
# Requires bitsandbytes + CUDA. On Jetson Orin, build bitsandbytes from source (sm_87).
from transformers import PaliGemmaForConditionalGeneration, PaliGemmaProcessor
from PIL import Image
import torch

repo = "lamao-ab/paligemma-blind-assist-qlora-merged-v1"
model = PaliGemmaForConditionalGeneration.from_pretrained(repo, device_map="cuda")
processor = PaliGemmaProcessor.from_pretrained(repo)

image = Image.open("example.jpg").convert("RGB")
prompt = "describe the scene for a blind person"  
inputs = processor(text=prompt, images=image, return_tensors="pt").to("cuda")
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=128)
print(processor.decode(out[0], skip_special_tokens=True))

Deployment note: genuine 4-bit (NF4, double-quantised) checkpoint for edge inference. Loading requires bitsandbytes + a CUDA GPU. On Jetson (L4T/JetPack), pip wheels do not work — build bitsandbytes from source with the device's compute capability (-DCOMPUTE_CAPABILITY=87 for Orin).

📄 License

Inherits the Gemma license from the base model.