abdur-rahman77/llama2-7b-qlora-guanaco

Llama-2-7B Chat - QLoRA adapter (Guanaco-1k)

LoRA adapter for NousResearch/Llama-2-7b-chat-hf, fine-tuned with QLoRA (4-bit NF4, double-quant, bf16) on mlabonne/guanaco-llama2-1k (1,000 multilingual instructions, Llama-2 chat format). Final training loss 1.47 (from 2.06), 1 epoch / 125 steps.

Training

• —LoRA r/alpha/dropout: 32/16/0.05, targets qproj,vproj
• —Effective batch 8, lr 0.0002 cosine, maxseqlen 256
• —pagedadamw32bit, Kaggle T4
• —transformers 4.56, trl 0.21, peft 0.15, bitsandbytes 0.47

Load the adapter on top of the base model with peft.PeftModel.from_pretrained.