digitalpathologybern/Lumen

Lumen

Lumen aligns frozen Virchow2 and BioMedBERT encoders using rank-4 LoRA adapters and projection heads into a shared 512-dimensional space.

This repository contains only the trained alignment parameters (2,984,961 parameters; 12.0 MB). The backbone weights are loaded from their original repositories.

Usage

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained("digitalpathologybern/Lumen", trust_remote_code=True).eval()

# images: (B, 3, 224, 224), preprocessed with the Virchow2 transform
tokens = model.tokenizer(["metastatic carcinoma in a lymph node",
                          "benign lymph node"],
                         padding=True, truncation=True, max_length=128,
                         return_tensors="pt")
with torch.no_grad():
    out = model(images=images, tokens=tokens)

img = torch.nn.functional.normalize(out["image_embeds"], dim=-1)
txt = torch.nn.functional.normalize(out["text_embeds"], dim=-1)
logits = out["logit_scale"] * img @ txt.T

images should have shape (B, 3, 224, 224) and be preprocessed using the Virchow2 transform.

encodeimage and encodetext can be used to run either encoder separately.

timm, transformers, and peft are required.

Both backbones are downloaded from their own repositories on the first call.

Licence

The 2,984,961 parameters in this repository are released under CC-BY-NC-4.0. The model also loads two third-party backbones, which are not redistributed here and carry their own terms.

The assembled model is non-commercial, a constraint that comes from Virchow2 and holds whatever these weights say. Read the backbone terms against your intended use rather than reading the tag above as covering the whole.

Adapting these weights further is permitted. Virchow2 additionally forbids derivatives of itself, so fine-tune the alignment, not the vision encoder.