Modelpublic

Likhith003/dpo-pairrm-lora-adapter

sourceHugging Faceapache-2.0updated 1y agoView on Hugging Face

0likes24downloads

Model Card

license: apache-2.0 language:

—en library_name: transformers tags:
—llama
—dpo
—preference-optimization
—PEFT
—instruction-tuning pipeline_tag: text-generation ---

DPO Fine-Tuned Adapter - PairRM Dataset

...

DPO Fine-Tuned Adapter - PairRM Dataset

🧠 Model

—Base: meta-llama/Llama-3.2-1B-Instruct
—Fine-tuned using TRL's DPOTrainer with the PairRM preference dataset (500 pairs)

⚙️ Training Parameters

Parameter	Value
Learning Rate	3e-5
Batch Size	4
Epochs	3
Beta (DPO regularizer)	0.1
Max Input Length	1024 tokens
Max Prompt Length	512 tokens
Padding Token	`eos_token`

📦 Dataset

—Source: pairrm_preferences.csv
—Size: 500 instructions with prompt, chosen, and rejected columns

📂 Output

—Adapter saved and uploaded as Likhith003/dpo-pairrm-lora-adapter