Modelpublic

yennj12/distilgpt2-lora-fact

sourceHugging Faceapache-2.0updated 25d agoView on Hugging Face

0likes31downloads

Model Card

distilgpt2 LoRA — "Fact:" answer style

A deliberately tiny LoRA adapter, trained as a teaching example. It teaches distilgpt2 one narrow behaviour: answer a Q: prompt with a single-line statement that begins with Fact:.

Trained on 8 hand-written examples in about 7 seconds on an Apple MPS laptop. It is not useful for anything real — it is useful for seeing what the moving parts of LoRA actually do.

What it does

Prompt	Base distilgpt2	With this adapter
`Q: What is LoRA?\nA:`	"LoRA is a very simple, simple, simple, simple…"	"Fact: LoRA freezes the base weights and trains two small matrices A and B."
`Q: What does a low-rank matrix do?\nA:` (held out)	"It's a matrix that is a matrix that is a matrix…"	"Fact: low-rank sets the weights of the edges, q_proj and v_proj first."

The first answer is memorised verbatim from the training set. The second prompt was never trained on: the Fact: style transferred, but the content is wrong — low-rank matrices do not "set the weights of the edges". That gap is the honest lesson of an 8-example fine-tune. LoRA transferred style, not knowledge.

Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("distilgpt2")
base = AutoModelForCausalLM.from_pretrained("distilgpt2")
model = PeftModel.from_pretrained(base, "yennj12/distilgpt2-lora-fact").eval()

inputs = tok("Q: What is LoRA?\nA:", return_tensors="pt")
print(tok.decode(model.generate(**inputs, max_new_tokens=24,
                                do_sample=False,
                                pad_token_id=tok.eos_token_id)[0]))

Call model.merge_and_unload() to fold the adapter into the base weights for zero inference overhead.

Training


Base model	`distilgpt2` (82M params)
Trainable params	147,456 (0.18%)
Adapter size	~584 KB, vs 313 MB for the base
Config	`r=8`, `lora_alpha=16` (scaling 2.0), `lora_dropout=0.05`
Target modules	`c_attn` — GPT-2's fused q/k/v projection, in all 6 blocks
Optimiser	AdamW, lr `2e-3`, 60 full-batch epochs
Loss	5.16 -> 0.51
Hardware	Apple Silicon (MPS), ~7 s
Seed	`torch.manual_seed(0)` — reproducible

The learning rate is ~100x a typical full fine-tuning LR. That is normal for LoRA: you are training freshly initialised matrices, not nudging pretrained weights.

Limitations

Everything about this model is a limitation. It was overfit on purpose to 8 sentences about LoRA, so it will state confident falsehoods on any prompt outside that set, and it inherits all of distilgpt2's biases underneath. Use it to learn how adapters work, not for generation.

Source

Training code and a from-scratch (no-PEFT) reimplementation of the same arithmetic: see the repository this adapter was trained from. Follows the Fine-Tuning with LoRA & QLoRA lesson from AI Engineering from Scratch.