Modelpublic

dasuntm/tinyllama-alpaca-qlora-v1

sourceHugging Facecc-by-nc-4.0updated 1mo agoView on Hugging Face

0likes9downloads

Model Card

TinyLlama-1.1B-Chat QLoRA (Alpaca)

LoRA adapter for TinyLlama/TinyLlama-1.1B-Chat-v1.0, trained with QLoRA (4-bit NF4 base, bf16 adapters) on a 2,000-row slice of Alpaca. A learning exercise, not a pro

Training


rank / alpha	16 / 32
target modules	q, k, v, o, gate, up, down
LR / schedule	2e-4, cosine, 3% warmup
effective batch	16
epochs	1
trainable params	12.6M (1.15%)

Loss was computed on assistant turns only, with EOS kept in the loss.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0")
model = PeftModel.from_pretrained(base, "dasuntm/tinyllama-alpaca-qlora-v1")
tok = AutoTokenizer.from_pretrained("dasuntm/tinyllama-alpaca-qlora-v1")

License

The base model is Apache 2.0, but Alpaca is CC BY-NC 4.0 (non-commercial) — it was
generated with OpenAI's API. This adapter inherits that restriction. Non-comm