sdoerrich97/mopet_dinov3_unified_blood_breast_derma_path

MoPET — unified (BloodMNIST, BreastMNIST, DermaMNIST, PathMNIST)

Official weights for MoPET: Parameter-Efficient Mixture-of-Experts for Unified Medical Image Classification (EMA4MICCAI 2026 Workshop). This repo holds the `unified` variant: a single MoPET model consolidating four heterogeneous MedMNIST+ classification tasks (Blood, Breast, Derma, Path) behind one shared, sparsely-routed PEFT expert pool. This same checkpoint is also used as the Breast-target auxiliary "booster" configuration in the paper.

MoPET unifies medical image classification across heterogeneous MedMNIST+ tasks by adapting a frozen DINOv3 ViT-B/16 backbone with a learned sparse top-k router over a heterogeneous pool of LoRA + BOFT experts injected into the attention qkv projections, trained with a DeepSeekMoE-style load-balancing loss and one lightweight classification head per dataset. Only the experts, routers, and heads are trained; the backbone stays frozen and is reconstructed from timm at load time.

Model Details

• —Developed by: Sebastian Doerrich, Daniel Würtinger (xAILab, University of Bamberg)
• —Funded by: Hightech Agenda Bayern (HTA) of the Free State of Bavaria, Germany
• —Model type: Mixture-of-Experts over parameter-efficient adapters (LoRA + BOFT) on a frozen Vision Transformer (DINOv3 ViT-B/16)
• —License: MIT

Model Sources

• —Paper: https://arxiv.org/abs/2607.29462
• —Repository: https://github.com/sdoerrich97/mopet
• —Package: https://pypi.org/project/mopet-moe/

Architecture and training

• —Head order (dataset id -> head): 0=BloodMNIST (8 classes), 1=BreastMNIST (2 classes), 2=DermaMNIST (7 classes), 3=PathMNIST (9 classes).

These are the lab's shared training-recipe values (see experiments/utils/training.py). The full multi-task loop is in the repo; a runnable demo is examples/training.ipynb.

Uses

Direct Use

Multi-task classification on the four MedMNIST+ datasets this checkpoint consolidates. A single forward pass routes the input through the shared expert pool and emits the per-dataset head logits.

Downstream Use

• —Feature extraction / probing: the frozen backbone plus routed experts can serve as a strong medical-image encoder for lightweight downstream heads.
• —Extending the pool: the mopet package supports adding a new dataset head and co-training it against the existing expert pool (see the repo's experiments/).

Out-of-Scope Use

• —Not a diagnostic device. These weights are a research artifact; the outputs must not be used for clinical diagnosis without expert validation and regulatory approval.
• —Non-medical or off-distribution images. The model is trained only on the datasets above at MedMNIST+ resolution; behavior on other modalities or natural images is undefined.

Bias, Risks, and Limitations

• —MedMNIST+ images are downsampled from their clinical sources, so fine-grained detail is lost relative to full-resolution acquisition.
• —Several source datasets are class-imbalanced; per-class performance varies and the headline accuracy can mask minority-class errors.
• —Reported numbers are in-distribution; robustness under acquisition/domain shift is not characterized by this checkpoint.
• —Top-k routing uses scatter/gather ops whose CUDA kernels are not fully deterministic; the package seeds best-effort (warn-only) determinism (see the repo's reproducibility.py).

How to Get Started with the Model

Installation

pip install mopet-moe

Input Requirements

• —Resolution: 3×256×256 RGB (MedMNIST+ images are loaded at 224 and upsampled to 256 for the DINOv3 backbone).
• —Format: PyTorch tensor (B, 3, 256, 256).
• —Normalization: applied by the package's preprocessing to match the DINOv3 backbone's expected statistics; see examples/inference.ipynb.

Inference

from mopet import create_model

model = create_model(weights="unified").eval()
# ... run inference; see the repo's examples/ notebooks.

The mopet package resolves this repo automatically via its PUBLISHED_MODELS map, downloading model.safetensors through huggingface_hub.

Evaluation

Metrics

Accuracy (ACC) and area under the ROC curve (AUC), per the paper.

Results

From the paper: on the four-dataset pool consolidated by this checkpoint, MoPET reaches 93.46% average accuracy, beating the best isolated per-dataset PEFT adapters (92.83%) trained on the same datasets individually. More broadly, across the paper's 12 MedMNIST+ datasets, parameter-efficient fine-tuning of the frozen backbone outperforms full end-to-end fine-tuning (86.50% -> 88.97% average accuracy). See the paper for the full per-dataset breakdown and the auxiliary-dataset "booster" analysis this checkpoint also participates in.

Citation

@article{doerrich2026mopet,
  title={MoPET: Parameter-Efficient Mixture-of-Experts for Unified Medical Image Classification},
  author={Sebastian Doerrich and Daniel W{\"u}rtinger and Francesco {Di Salvo} and Shyam Nandan Rai and Christian Ledig},
  year={2026},
  eprint={2607.29462},
  archivePrefix={arXiv},
  primaryClass={eess.IV},
  url={https://arxiv.org/abs/2607.29462},
}

Model Card Contact

For questions or issues, please open an issue in the GitHub repository.