Modelpublic

swarajsonawane4/scifact-phi3-lora

sourceHugging Faceupdated 5mo agoView on Hugging Face

1likes4downloads

Model Card

SciFact Phi-3 Mini LoRA Adapter

LoRA adapter fine-tuned on the SciFact dataset for biomedical claim verification. Given a scientific claim and retrieved evidence, the model produces structured verdicts (SUPPORTED / REFUTED / INSUFFICIENT) with citations.

Use Case

This adapter sits on top of a RAG pipeline that retrieves biomedical evidence. The fine-tuned model generates grounded answers with explicit document citations, suitable for fact-checking applications.

Training Details

—Base model: microsoft/Phi-3-mini-4k-instruct (3.8B params)
—Method: LoRA (Parameter-Efficient Fine-Tuning)
—Rank: 8, Alpha: 16, Dropout: 0.05
—Target modules: qkvproj, oproj
—Quantization: 4-bit NF4 via bitsandbytes (for training)
—Training data: 759 examples from SciFact train split
—Evaluation data: 50 held-out examples
—Epochs: 5
—Effective batch size: 16
—Learning rate: 2e-4 (cosine decay)
—Hardware: Tesla T4 x2 (Kaggle)

Results

Epoch	Training Loss	Validation Loss
1	0.527	0.687
2	0.406	0.671
3	0.487	0.662
4	0.415	0.658
5	0.487	0.659

Generalization gap: 0.17 (healthy, no overfitting).

Usage

Load with PEFT on top of the base model:

from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel import torch

BASE = "microsoft/Phi-3-mini-4k-instruct" ADAPTER = "swarajsonawane4/scifact-phi3-lora"

tokenizer = AutoTokenizer.frompretrained(BASE) basemodel = AutoModelForCausalLM.frompretrained( BASE, torchdtype=torch.bfloat16, devicemap="auto" ) model = PeftModel.frompretrained(base_model, ADAPTER) model.eval()

Output Format

Verdict: SUPPORTED | REFUTED | INSUFFICIENT

Brief justification citing [D0], [D1], etc.

Author

Swaraj Sudhakar Sonawane - MSc. Digital Engineering, Bauhaus University Weimar