stanfordmimi/Merlin
3611k
1---2license: mit3language:4- en5library_name: merlin6pipeline_tag: text-to-image7---8 9# Merlin: A Computed Tomography Vision–Language Foundation Model and Dataset10 11[](https://doi.org/10.1038/s41586-026-10181-8) [](https://pypi.org/project/merlin-vlm/)12 13Merlin is a 3D VLM for computed tomography that leverages both structured electronic health records (EHR) and unstructured radiology reports for pretraining. The huggingface repository here provides the model weights and an example image file (Nature 2026).14 15[[💻 Github](https://github.com/StanfordMIMI/Merlin)] [[📄 Nature Paper](https://www.nature.com/articles/s41586-026-10181-8)]16 17## ⚡️ Installation18 19To install Merlin, you can simply run:20 21```python22pip install merlin-vlm23```24 25For an editable installation, use the following commands to clone and install this repository.26```python27git clone https://github.com/StanfordMIMI/Merlin.git28cd Merlin29pip install -e .30```31 32For usage instructions, please visit the github [repository](https://github.com/StanfordMIMI/Merlin).33 34### 📁 Project Structure:35 36```37.38├── README.md39├── i3_resnet_clinical_longformer_best_clip_04-02-2024_23-21-36_epoch_99.pt <Merlin weights>40├── image1.nii.gz <Sample Image>41├── resnet_gpt2_best_stanford_report_generation_average.pt <Merlin Radiology Report Generation Weights>42├── resnet_clinical_longformer_five_year_disease_prediction <Five Year Disease Prediction Weights>43├── nnUNetTrainerMerlin__nnUNetPlans__3d_fullres <nnUNet File>44```45 46## 📎 Citation47If you find this repository useful for your work, please cite the cite the [Nature paper](https://www.nature.com/articles/s41586-026-10181-8):48 49```bibtex50@article{blankemeier_kumar2026merlin,51 author = {Blankemeier, Louis and Kumar, Ashwin and Cohen, Joseph Paul and Liu, Jiaming and Liu, Longchao and Van Veen, Dave and Gardezi, Syed Jamal Safdar and Yu, Hongkun and Paschali, Magdalini and Chen, Zhihong and Delbrouck, Jean-Benoit and Reis, Eduardo and Holland, Robbie and Truyts, Cesar and Bluethgen, Christian and Wu, Yufu and Lian, Long and Jensen, Malte Engmann Kjeldskov and Ostmeier, Sophie and Varma, Maya and Valanarasu, Jeya Maria Jose and Fang, Zhongnan and Huo, Zepeng and Nabulsi, Zaid and Ardila, Diego and Weng, Wei-Hung and Amaro Junior, Edson and Ahuja, Neera and Fries, Jason and Shah, Nigam H. and Zaharchuk, Greg and Willis, Marc and Yala, Adam and Johnston, Andrew and Boutin, Robert D. and Wentland, Andrew and Langlotz, Curtis P. and Hom, Jason and Gatidis, Sergios and Chaudhari, Akshay S.},52 title = {Merlin: a computed tomography vision-language foundation model and dataset},53 journal = {Nature},54 year = {2026},55 doi = {10.1038/s41586-026-10181-8},56 url = {https://doi.org/10.1038/s41586-026-10181-8}57}58```