CoolFace
Modelpublic

iamaber/mistral-7b-pubmedqa-adapter

sourceHugging Faceapache-2.0updated 6mo agoView on Hugging Face
0likes
Model Card

pubmedqa-loraplus Adapter Model

Overview

This adapter artifact fine-tunes mistralai/Mistral-7B-Instruct-v0.3 on qiaojin/PubMedQA / pqa_labeled using LoRA+.

Training Setup

FieldValue
Train examples900
Eval examples100
Epochs3
Train batch size4
Eval batch size4
Gradient accumulation4
Learning rate5e-05
Best eval loss0.5188
Latest eval loss0.5188
Latest train loss0.4810
Train runtime (s)566.5676
Global step171

Evaluation Summary

MetricValue
PubMedQA accuracy0.4500
PubMedQA macro F10.2069
PubMedQA weighted F10.2793
PubMedQA samples100
Medical MMLU accuracy0.1600
Medical MMLU samples50

Medical MMLU Subject Breakdown

SubjectAccuracyCorrectTotal
anatomy0.1600850
clinical_knowledge0.000000
college_medicine0.000000
medical_genetics0.000000
professional_medicine0.000000
virology0.000000

PubMedQA Confusion Matrix

Actual \ Predictedyesnomaybe
yes4500
no4000
maybe1500