robi913/medsiglip-retinal-oct-lora

Model Card for MedSigLIP-OCT: Dual-Prompt Cross-Attention Fusion

This model is a fine-tuned adaptation of google/medsiglip-448 for Retinal Optical Coherence Tomography (OCT) analysis. It adds a Cross-Attention Fusion module that works around the 64-token limit of the SigLIP text encoder, so that detailed, multi-part clinical reports can be used for image-text alignment.

The model is a research prototype. It is not a medical device and it has not been clinically validated.

Model Details

Model Description

The SigLIP-family text encoder used by MedSigLIP reads at most 64 tokens. In OCT reporting, a single description has to cover both the anatomical layer structure and the pathological biomarkers, which usually takes 180 to 256 tokens. Truncation would discard part of this content. (Newer VLMs with larger text encoders do not have this limit. We keep MedSigLIP because it already provides a medical image-text alignment.)

To work within the limit, each generated report is split into two sub-prompts: prompt_a (structural profile) and prompt_b (pathological profile). Both are encoded separately and merged by a Cross-Attention Fusion module into one text embedding ($t_{final}$) that lies on the same unit sphere as the image embedding ($v$).

• —Developed by: Robert-Emanuel Ardelean (Technical University of Cluj-Napoca)
• —Shared by: Robert-Emanuel Ardelean
• —Model type: Vision-Language Model (multi-task)
• —Language(s) (NLP): English
• —License: Health AI Developer Foundations (inherited from the base model)
• —Finetuned from model: google/medsiglip-448

Model Sources

• —Code: https://github.com/Robi-913/Licenta2026ArdeleanRobertEmanuel
• —Base model: `google/medsiglip-448`
• —Paper: "Text Embedding Cross Fusion for Overcoming Token Limits in Vision Language Models" (accepted for CSSC UTCN 2026).
• —Paper: "Domain Adaptation of Vision-Language Models for Retinal OCT Analysis via Multi-Task Latent Alignment and Cross-Attention Fusion" (accepted for ICCP 2026).

Files in this repository

Both files are pickled PyTorch checkpoints. Only load them from a source you trust.

Uses

Direct Use

The model provides aligned image and text embeddings for retinal OCT scans and supports:

• —Cross-modal retrieval: image-to-text (I2T) and text-to-image (T2I). Retrieval is evaluated at the diagnosis level (see Evaluation).
• —Classification: AMD, DME, DRUSEN or NORMAL. The reported accuracy is obtained with the classification head trained after the joint training (see Training). Zero-shot use of the original MedSigLIP reaches only 25.8% accuracy on this test set.
• —Severity estimation: a continuous 0-100 score that follows the biomarker-derived index described below.
• —Biomarker detection: 9 biomarkers, with the separate Biomarker_Detection.pth checkpoint.

Downstream Use

The model can be combined with explainability tools (for example EigenCAM, computed with SVD on ViT activations) and uncertainty estimation (for example Monte Carlo Dropout) for research on retinal pathologies.

Out-of-Scope Use

Research use only. The severity score and the biomarker heads are not medically validated, and the model must not be used for patient diagnosis or for autonomous clinical decisions without strict clinical validation.

Bias, Risks, and Limitations

• —Synthetic reports: the training texts were generated by MedGemma 27B from the images and annotations, then split by Gemini Flash-Lite. They were not reviewed by clinicians.
• —Severity score: the score is our own construction, made with medical input but not a clinically validated scale. It is built from a base value per disease category, weights per biomarker, the lesion area and a log-scaled lesion count. Healthy scans without physician-drawn boxes receive a small value in the range 0-8. Lesion counts come from physician annotations where available, otherwise from YOLO detections verified and corrected by MedGemma.
• —Severity propagation: because the score contains a base value that depends on the disease, severity is closely tied to classification. A misclassification (for example a normal scan predicted as AMD) also distorts the predicted severity.
• —Overconfidence: in our Monte Carlo Dropout run (20 passes, 748 test images, an earlier version of the model), the mean confidence was 97.8% for an accuracy of 83.2%, with an Expected Calibration Error of 0.32. Raw confidence values should not be trusted without calibration (for example temperature scaling).
• —Diagnosis-level retrieval: Recall@K counts a hit when a retrieved item has the same diagnosis as the query. It is not exact image-report matching and it is closely related to classification accuracy.
• —Small classes: the test set contains only 8 Drusen images, so Drusen results are very unstable. AMD is a heterogeneous class (from early drusen to geographic atrophy), and AMD versus Normal is the main source of confusion.
• —Weak labels: part of the biomarker annotations used during dataset expansion are silver labels (YOLOv12e detections with a 0.25 confidence threshold). The biomarker heads themselves were trained on physician-verified annotations only.
• —Single run: all results come from one training run, without confidence intervals.

Recommendations

Use the model only as an assistive research tool with a human in the loop.

How to Get Started with the Model

The fusion module and the task heads are custom classes, so the source code is required. The main checkpoint is loaded into MedSigLIPMultiTask from the GitHub repository.

git clone https://github.com/Robi-913/Licenta_2026_Ardelean_Robert_Emanuel
cd Licenta_2026_Ardelean_Robert_Emanuel
pip install -r requirements.txt huggingface_hub
huggingface-cli login   # google/medsiglip-448 is gated: accept its terms on the Hub first

Run the following from the repository root (it imports src.model.medsiglip):

import torch
import torch.nn.functional as F
from huggingface_hub import hf_hub_download
from PIL import Image, ImageFilter
from transformers import AutoProcessor

from src.model.medsiglip import MedSigLIPMultiTask

BASE_ID = "google/medsiglip-448"
REPO_ID = "robi913/medsiglip-retinal-oct-lora"
device = "cuda" if torch.cuda.is_available() else "cpu"

# ---- 1. Load the main model ------------------------------------------------
processor = AutoProcessor.from_pretrained(BASE_ID)

ckpt_path = hf_hub_download(REPO_ID, "MedSigLip_finetune_LoRA_linear_probe.pth")
ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)  # pickled: trusted source only
state = ckpt.get("model", ckpt)

classes = ckpt.get("classes", ["AMD", "DME", "DRUSEN", "NORMAL"])
cls_hidden = state["classification_head.1.weight"].shape[0]  # 512 = probed head

model = MedSigLIPMultiTask(BASE_ID, n_classes=len(classes), cls_hidden=cls_hidden)
missing, unexpected = model.load_state_dict(state, strict=False)
print(f"missing keys: {len(missing)} | unexpected keys: {len(unexpected)}")  # both should be 0 or tiny
model = model.to(device).eval()


# ---- 2. Preprocessing (same as in training) --------------------------------
def prep_image(path):
    img = Image.open(path).convert("RGB").filter(ImageFilter.GaussianBlur(radius=0.5))
    return processor(images=img, return_tensors="pt")["pixel_values"].to(device)


def tokenize(text):
    enc = processor.tokenizer(
        text, padding="max_length", truncation=True, max_length=64, return_tensors="pt"
    )
    ids = enc["input_ids"]
    mask = enc["attention_mask"] if "attention_mask" in enc else torch.ones_like(ids)
    return ids.to(device), mask.to(device)


# ---- 3. One scan + its two sub-prompts (each at most 64 tokens) ------------
pixel_values = prep_image("sample_oct.jpg")
prompt_a = "Structural profile: description of the retinal layers, thickness and layer integrity ..."
prompt_b = "Pathological profile: description of the lesions and biomarkers present ..."
ids_a, mask_a = tokenize(prompt_a)
ids_b, mask_b = tokenize(prompt_b)

with torch.no_grad():
    img_emb, emb_a, emb_b, fused_emb, logit_scale, severity, logits = model(
        pixel_values, ids_a, mask_a, ids_b, mask_b
    )

# Classification (from the image only)
probs = logits.softmax(dim=-1)[0]
print({c: round(float(p), 3) for c, p in zip(classes, probs)})

# Severity index (network output is in [0, 1], multiply by 100)
print(f"severity: {float(severity[0]) * 100:.1f} / 100")

# Image <-> text similarity (all embeddings are L2-normalized)
print("image vs prompt_a :", float(img_emb @ emb_a.T))
print("image vs prompt_b :", float(img_emb @ emb_b.T))
print("image vs fused    :", float(img_emb @ fused_emb.T))

For retrieval over many scans, stack the img_emb and fused_emb vectors of all examples and compute img_embs @ fused_embs.T. Rank each row (I2T) or each column (T2I).

Biomarker detection (optional)

from src.model.medsiglip import BiomarkerHeadsV5

bm_path = hf_hub_download(REPO_ID, "Biomarker_Detection.pth")
bm_ckpt = torch.load(bm_path, map_location="cpu", weights_only=False)

bm_hidden = bm_ckpt["model"]["backbone.classification_head.1.weight"].shape[0]
bm_backbone = MedSigLIPMultiTask(BASE_ID, cls_hidden=bm_hidden)
bm_model = BiomarkerHeadsV5(backbone=bm_backbone, n_biomarkers=len(bm_ckpt["biomarkers"]))
bm_model.load_state_dict(bm_ckpt["model"], strict=False)
bm_model = bm_model.to(device).eval()

with torch.no_grad():
    bm_probs = torch.sigmoid(bm_model(pixel_values))[0]

for name, p, thr in zip(bm_ckpt["biomarkers"], bm_probs, bm_ckpt["thresholds"]):
    print(f"{name:<24} p={float(p):.2f}  {'present' if float(p) >= thr else 'absent'}")

Training Details

Training Data

The model was fine-tuned on the OCT5k dataset (4,596 images, 4 classes: AMD, DME, Drusen, Normal). Splits (splits_v3) were built at the patient level to avoid leakage between training and test data.

Training Procedure

Preprocessing

• —Biomarker detection: YOLOv12e generated bounding boxes for images without manual annotations.
• —Report generation: MedGemma 27B (4-bit) generated 180-256 token reports from the OCT image, layer information and bounding boxes.
• —Prompt splitting: Gemini Flash-Lite split each report into prompt_a (anatomy) and prompt_b (pathology), each within the 64-token limit.
• —Images: RGB, light Gaussian blur (radius 0.5), 448x448 input through the MedSigLIP processor.

Training Hyperparameters

• —Training regime: mixed-precision multi-task learning, effective batch size 64 (8 images x 8 gradient-accumulation steps), up to 35 epochs (3 warmup epochs, cosine annealing), gradient clipping and early stopping (patience 10).
• —Optimizer: AdamW, learning rate 1.5e-4 for the LoRA parameters and 1e-4 for the fusion module and heads, weight decay 0.01.
• —Fine-tuning strategy: the pretrained weights stay frozen and only the LoRA adapters are trained ($r=16$, $\alpha=32$, dropout 0.05, on the q_proj, k_proj, v_proj and out_proj layers of both the vision and the text encoders).
• —Sampling: images with bounding-box annotations are oversampled 3x.
• —Multi-task loss weights:
• —SigLIP contrastive loss: 2.0 (average of image-prompt_a, image-prompt_b and image-fused-text terms)
• —Disease classification (cross-entropy): 0.3
• —Severity regression (Smooth L1): 0.2
• —Classification head: after joint training, a deeper MLP (512 to 256 hidden units) was trained on the frozen, normalized image embeddings, with backbone, fusion and severity head frozen. This step does not change the embeddings, so retrieval is unaffected. The classification accuracy below is measured after this step.
• —Biomarker heads: trained separately on the frozen fine-tuned backbone, using physician-verified annotations only, with Focal Loss ($\alpha=0.25$, $\gamma=2.0$).

Evaluation

Testing Data, Factors & Metrics

Testing Data

Patient-disjoint test split (splits_v3) with 748 OCT images (339 AMD, 191 DME, 8 Drusen, 210 Normal).

Metrics

• —Classification accuracy and macro F1.
• —Average Recall@1 (retrieval): mean of I2T and T2I Recall@1. A query counts as a hit if at least one of the top-K results has the same diagnosis as the query (diagnosis-level retrieval).
• —Severity MAE: mean absolute error in points on the 0-100 severity index.

Results

"-" means the model has no such output or the value was not measured.

Per-class results of this model:

Summary

The fine-tuned model reaches the best value in every column of the comparison above. The ResNet18 baseline was trained from scratch on a small training set and tends to predict Normal, so it is a weak baseline. No ablations (single prompt versus fused prompts, classification-only fine-tuning, ImageNet-pretrained CNN) are reported yet.

Technical Specifications

Model Architecture and Objective

• —Vision encoder: ViT encoder of medsiglip-448 with LoRA adapters.
• —Text encoder: MedSigLIP text encoder with LoRA adapters (pretrained weights frozen). Each sub-prompt is encoded separately (at most 64 tokens).
• —Fusion module: two multi-head attention blocks (4 heads), one in each direction (structural attends to pathological and the reverse), a learned sigmoid gate that mixes the two outputs per dimension, residual connections from both prompt embeddings, layer normalization, a small residual MLP and L2 normalization. Each sub-prompt is a single pooled vector, so every attention block sees a sequence of length one and its softmax equals 1. In practice each block returns a learned projection of the other prompt, and the gate does the mixing.
• —Task heads: MLP heads for 4-way classification and for severity regression (both on the pooled image feature), and 9 independent MLP heads for biomarker detection (Fluid, Geographic atrophy, PR layer disruption, Soft drusen PED, Reticular drusen, Hyperfluorescent spots, Soft drusen, Hard drusen, Choroidal folds).

Acknowledgements: Developed at the Computer Science Department, Technical University of Cluj-Napoca (UTCN). This work was supported in part by the project "Romanian Hub for Artificial Intelligence - HRIA", Smart Growth, Digitization and Financial Instruments Program, MySMIS no. xxxxx. Thanks to the creators of the OCT5k dataset and to Google for the MedSigLIP and MedGemma foundation models.