CoolFace
Modelpublic

iamaber/mistral-7b-pubmedqa-lora-plus

sourceHugging Faceapache-2.0updated 6mo agoView on Hugging Face
0likes12downloads
Model Card

pubmedqa-loraplus Model

Overview

This merged artifact fine-tunes mistralai/Mistral-7B-Instruct-v0.3 on qiaojin/PubMedQA / pqa_labeled using LoRA+.

Training Setup

FieldValue
Train examples900
Eval examples100
Epochs3
Train batch size4
Eval batch size4
Gradient accumulation4
Learning rate5e-05
Best eval loss0.5188
Latest eval loss0.5188
Latest train loss0.4810
Train runtime (s)566.5676
Global step171

Evaluation Summary

MetricValue
PubMedQA accuracy0.4500
PubMedQA macro F10.2069
PubMedQA weighted F10.2793
PubMedQA samples100
Medical MMLU accuracy0.1600
Medical MMLU samples50

Medical MMLU Subject Breakdown

SubjectAccuracyCorrectTotal
anatomy0.1600850
clinical_knowledge0.000000
college_medicine0.000000
medical_genetics0.000000
professional_medicine0.000000
virology0.000000

PubMedQA Confusion Matrix

Actual \ Predictedyesnomaybe
yes4500
no4000
maybe1500