Modelpublic

brooksmz4/llama-3.1-8b-finance-dora

sourceHugging Facellama3updated 7mo agoView on Hugging Face

0likes12downloads

Model Card

llama-3.1-8b-finance-dora

This model is a DoRA (Weight-Decomposed Low-Rank Adaptation) fine-tuned version of unsloth/Meta-Llama-3.1-8B-Instruct trained on a combined financial and general instruction dataset to function as a conversational financial advisor. Only ~1–2% of total parameters were updated during training, with base model weights kept frozen. Trained using TRL SFTTrainer via the Unsloth library.

Quick Start

python

from transformers import pipeline

question = "What is dollar cost averaging?"
generator = pipeline("text-generation", model="Mohammedmz4/llama-3.1-8b-finance-dora", device="cuda")
output = generator([{"role": "user", "content": question}], max_new_tokens=256, return_full_text=False)[0]
print(output["generated_text"])

Training Details

Parameter	Value
Base model	unsloth/Meta-Llama-3.1-8B-Instruct
Method	DoRA (Weight-Decomposed Low-Rank Adaptation)
Precision	BF16
Rank (r)	32
Alpha (α)	64
Target modules	q, k, v, o, gate, up, down (proj)
Adapter dropout	0.05
Trainable parameters	~1–2% of total
Learning rate	2×10⁻⁴
Epochs	1
Effective batch size	32
Optimiser	Paged AdamW 8-bit
Gradient clipping	Max norm 0.3
LR schedule	Cosine decay
Warmup steps	100
Weight decay	0.01
Saved	Adapter weights + tokeniser (~400–500 MB)

Evaluation Results

Metric	Baseline	DoRA
ROUGE-1	0.3379	0.4003
ROUGE-2	0.1436	0.2000
ROUGE-L	0.2490	0.3236
Perplexity ↓	7.699	6.322
BERTScore F1	0.5636	0.6791
ConvFinQA Accuracy	0.0933	0.1240

DoRA outperformed the untuned baseline across every metric and outperformed Full Fine-Tuning (FFT) on all metrics including ConvFinQA, where FFT degraded below baseline due to catastrophic forgetting.

Framework Versions

—PEFT: 0.18.1
—TRL: 0.24.0
—Transformers: 4.57.6
—Pytorch: 2.10.0
—Datasets: 4.3.0
—Tokenizers: 0.22.2
—Unsloth: latest

Citations

bibtex

@misc{vonwerra2022trl,
    title        = {{TRL: Transformer Reinforcement Learning}},
    author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallo{\'e}dec},
    year         = 2020,
    journal      = {GitHub repository},
    publisher    = {GitHub},
    howpublished = {\url{https://github.com/huggingface/trl}}
}

@misc{liu2024dora,
    title        = {DoRA: Weight-Decomposed Low-Rank Adaptation},
    author       = {Shih-Yang Liu and Chien-Yi Wang and Hongxu Yin and Pavlo Molchanov and Yu-Chiang Frank Wang and Kwang-Ting Cheng and Min-Hung Chen},
    year         = {2024},
    eprint       = {2402.09353},
    archivePrefix= {arXiv},
    primaryClass = {cs.CL}
}