X-iZhang/libra-llava-med-v1.5-mistral-7b

LLaVA-Med v1.5 (based on Mistral-7B-Instruct-v0.2)

LLaVA-Med (Large Language and Vision Assistant for bioMedicine) is an open-source large vision-language model adapted for biomedical applications. Built upon LLaVA and enhanced through curriculum learning, LLaVA-Med is fine-tuned specifically for open-ended biomedical question answering tasks.

This release aims to support research reproducibility for the corresponding paper, which demonstrates improved performance on biomedical VQA benchmarks such as PathVQA and VQA-RAD.

📌 Note: For original model weights, refer to microsoft/llava-med-v1.5-mistral-7b.

📃 Original paper: LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day.

🔬 Experimental Usage in Libra's repo

This model checkpoint is intended for experimental use and can be tested directly within the **Libra repository**.

Key Modification

To enable the re-trained vision encoder during inference, ensure the following configuration is applied:

"unfreeze_mm_vision_tower": true

📚 Learn More

For a deeper dive into the methodology, theoretical insights, and performance benchmarks of the Libra framework, please see the following resources:

• —🔗 Project Website: Libra v1.0
• —📄 Paper: arXiv:2411.19378
• —💻 Code Repository: X-iZhang/Libra (GitHub)
• —📷 Related Project: CCD – Clinical Change Detection; see technical details in the paper here.

License

mistralai/Mistral-7B-Instruct-v0.2 license.