medarc/spectra-h-optimus-0-lora

SPECTRA LoRA - H-optimus-0

A LoRA adapter that makes bioptimus/H-optimus-0 robust to changes in slide acquisition -- scanner, stain, and centre. It is trained contrastively on registered PLISM tiles, where the same physical tissue location is imaged under many scanner/stain conditions, so the objective is to pull matched conditions of one tile together while pushing different tiles apart. The base model's weights are untouched; only a rank-32 LoRA delta on the attention and MLP projections is learned and released.

Base model

The base model bioptimus/H-optimus-0 is gated on the Hugging Face Hub. You must request and be granted access, and be authenticated (huggingface-cli login), before the snippet below can download the base weights. The adapter alone is useless without them.

This adapter was trained and evaluated against that exact revision. Applying it to a different revision of the base model is untested.

Seeds

Three independent training seeds are shipped as subfolders. They are not an ensemble -- pick one, or report the spread across all three.

Seed label remapping. The published folder seed2/ was trained with seed = 3, not 2 (there is no s2 run for this backbone). The folder names are normalised to seed0/seed1/seed2 for consistency across the release; the original training seed value is recorded in each folder's training_config.json under "seed".

Usage

import torch
import torchvision.transforms as T
from PIL import Image

import timm
from peft import PeftModel

# Load the pinned base weights via timm's hub path. The architecture kwargs below are
# passed explicitly and override anything in the repo config; do NOT build this model
# from the bare architecture name "vit_giant_patch14_reg4_dinov2" -- timm's built-in default config
# for that name is a different model.
base = timm.create_model(
    "hf-hub:bioptimus/H-optimus-0@b145cc1e6c6b30d3251aa8b1f844e6974188a743",
    pretrained=True,
    img_size=224, init_values=1e-5, dynamic_img_size=False,
    num_classes=0, global_pool="token",
)

model = PeftModel.from_pretrained(base, "medarc/spectra-h-optimus-0-lora", subfolder="seed0")
model = model.merge_and_unload()          # fold LoRA into the base weights
model = model.float().eval().cuda()

Preprocessing -- this must match exactly:

tf = T.Compose([
    T.Resize(256, interpolation=T.InterpolationMode.BICUBIC),
    T.CenterCrop(224),
    T.ToTensor(),
    T.Normalize(mean=(0.707223, 0.578729, 0.703617), std=(0.211883, 0.230117, 0.177517)),
])

Forward pass and readout:

img = Image.open("tile.png").convert("RGB")
x = tf(img).unsqueeze(0).cuda()

with torch.inference_mode():
    h = model.forward_features(x)                  # (B, P + N, C)
    feat = h[:, 0]                                 # (B, 1536)

Readout: CLS token alone: h[:, 0] (equivalently the model output under global_pool="token", num_classes=0) Embedding dimension: 1536 Token layout: 256 spatial tokens; embeddim 1536; numprefix_tokens = 5 (1 CLS + 4 registers). Patch/dense tokens begin at index 5.

Nothing else from the training run is needed or released. The contrastive projector heads and the pooling head were training-only machinery and are deliberately not part of this repository.

WARNINGS

1. Normalisation is H&E-specific, NOT ImageNet.

mean=(0.707223, 0.578729, 0.703617), std=(0.211883, 0.230117, 0.177517), read from the checkpoint's own pretrained_cfg. timm's built-in config for vit_giant_patch14_reg4_dinov2 is DINOv2's LVD-142M config, whose ImageNet-ish statistics have the same shape and raise no warning - they simply cost accuracy on every downstream task. Always resolve the transform from the checkpoint's config.

2. img_size=224 is mandatory.

The architecture default is 518, whose pos_embed is (1, 1369, 1536) and will not load against this checkpoint's (1, 256, 1536).

3. Resize is 256 then CenterCrop 224 (crop_pct 0.875), not a direct 224 resize.

The pretrained_cfg carries no crop_pct and no interpolation, so timm's defaults (bicubic, crop_pct=0.875) apply.

Results

Base model versus base model + this adapter. Values are read from the SPECTRA paper's tables. n = 3 seeds; the interval is mean +/- 2SD across those three seeds, quoted verbatim from the paper's tables (which already report 2SD).

Higher is better on every row. Base-model numbers are single deterministic evaluations of the frozen base and carry no seed spread.

Training

Checkpoints were written every 50 steps; only the 1-SE-selected step per seed is released. Because selection is per seed, the three seeds of a backbone need not sit at the same step -- and because the selected steps fall inside the 200-step warmup, the released checkpoints are un-annealed.

Full run hyper-parameters are in each seed folder's training_config.json.

`training_config.json` encoding note. The training code writes same_core_logit_bias_mean as negative infinity, which Python's json module emits as the bare token -Infinity. That token is not valid JSON and is rejected by JSON.parse and most non-Python parsers. In the released files it is encoded as the string `"-Infinity"` so that the file parses everywhere. Read it back as float("-inf"). This is the only edit made to the training configuration.

Citation

@inproceedings{spectra2026,
  title  = {SPECTRA: cross-acquisition robustness for pathology foundation models},
  author = {TODO: author list},
  year   = {2026},
  note   = {TODO: confirm venue and year before citing -- submitted to NeurIPS 2026},
  url    = {https://github.com/TODO-org/spectra}
}

Code: SPECTRA on GitHub (TODO: fill in the canonical repository URL before publishing).

Licence

See LICENSE in this repository. The adapter weights are MIT-licensed; the base model carries its own separate licence, which you must also comply with.