Modelpublic

ThakrePranjal/pharma-tinyllama-unsloth-stage1-lora

sourceHugging Faceupdated 3mo agoView on Hugging Face

0likes7downloads

Model Card

Pharma TinyLlama — Stage 1 LoRA (Domain Pretraining) — Unsloth

Stage 1 LoRA adapter from the Unsloth 3-stage pharma fine-tuning pipeline. Trained on raw pharmaceutical PDF text paragraphs using SFTTrainer with Unsloth's 4-bit fast patching.

Training details

Param	Value
Base model	`unsloth/tinyllama-bnb-4bit`
Trainer	`trl.SFTTrainer` (Unsloth patched)
Data	Raw PDF paragraphs (9 records, packing=True)
Max steps	30
Learning rate	2e-4
LoRA r	16
LoRA alpha	32
Peak VRAM	0.98 GB
Train time	161s

Usage

python

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/tinyllama-bnb-4bit",
    max_seq_length=512,
    load_in_4bit=True,
)

from peft import PeftModel
model = PeftModel.from_pretrained(model, "ThakrePranjal/pharma-tinyllama-unsloth-stage1-lora")

Pipeline

unsloth/tinyllama-bnb-4bit
    └── Stage 1 SFT (THIS ADAPTER) → merged → [ThakrePranjal/pharma-tinyllama-unsloth-stage1-merged]
            └── Stage 2 SFT        → merged → [ThakrePranjal/pharma-tinyllama-unsloth-stage2-merged]
                    └── Stage 3 DPO → merged → [ThakrePranjal/pharma-tinyllama-unsloth-final]