Clover-Hill/MemoryDecoder-Llama-biomed

Model Description

This Memory Decoder model is trained on the Biomedical domain and can be adapted to enhance any model in the Llama3, Llama3.1, and Llama3.2 families.

[!IMPORTANT] These Llama models are initialized from Qwen models with the embedding layer adapted to fit the Llama tokenizer. This enables efficient cross-model family knowledge transfer.

Paper: Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models

GitHub: https://github.com/LUMIA-Group/MemoryDecoder

Training & Evaluation Data

Biomedical Domain Dataset: mimic_iii_diagnosis_anonymous

Test Split: MemoryDecoder-domain-data

Performance Results

Llama3 Family

Llama3.1 Family

Llama3.2 Family

Perplexity scores on Biomedical domain test set. Lower is better.

Citation

@article{cao2025memory,
  title={Memory decoder: A pretrained, plug-and-play memory for large language models},
  author={Cao, Jiaqi and Wang, Jiarui and Wei, Rubin and Guo, Qipeng and Chen, Kai and Zhou, Bowen and Lin, Zhouhan},
  journal={arXiv preprint arXiv:2508.09874},
  year={2025}
}

Contact

For questions and support: maximus.cao@outlook.com