argilla/phi2-lora-distilabel-intel-orca-dpo-pairs
<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->
phi2-lora-quantized-distilabel-intel-orca-dpo-pairs
This model is a fine-tuned version of microsoft/phi-2 on distilabel-intel-orca-dpo-pairs. The full training notebook can be found here.
It achieves the following results on the evaluation set:
- Loss: 0.4537
- Rewards/chosen: -0.0837
- Rewards/rejected: -1.2628
- Rewards/accuracies: 0.8301
- Rewards/margins: 1.1791
- Logps/rejected: -224.8409
- Logps/chosen: -203.2228
- Logits/rejected: 0.4773
- Logits/chosen: 0.3062
Model description
The adapter was fine-tuned on a Google Colab A100 GPU using DPO and the distilabel-intel-orca-dpo-pairs. In order to scale LoRa approached for LLMs, I recommend looking at predibase/lorax.
You can play around with the model shown below. We load the LoRa adapter and bitsnbytes config (only when CUDA is available).
import torch
import torch
from transformers import (
AutoModelForCausalLM,
AutoTokenizer,
BitsAndBytesConfig
)
from peft import PeftModel
# template used for fine-tune
# template = """\
# Instruct: {instruction}\n
# Output: {response}"""
if torch.cuda.is_available():
device = torch.device("cuda")
print(f"Using {torch.cuda.get_device_name(0)}")
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type='nf4',
bnb_4bit_compute_dtype='float16',
bnb_4bit_use_double_quant=False,
)
elif torch.backends.mps.is_available():
device = torch.device("mps")
bnb_config = None
else:
device = torch.device("cpu")
bnb_config = None
print("No GPU available, using CPU instead.")
config = PeftConfig.from_pretrained("davidberenstein1957/phi2-lora-quantized-distilabel-intel-orca-dpo-pairs")
model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2", torch_dtype=torch.float16, quantization_config=bnb_config)
model = PeftModel.from_pretrained(model, "davidberenstein1957/phi2-lora-quantized-distilabel-intel-orca-dpo-pairs").to(device)
prompt = "Instruct: What is the capital of France? \nOutput:""
inputs = tokenizer(prompt, return_tensors="pt", return_attention_mask=False)
outputs = model.generate(**inputs)
text = tokenizer.batch_decode(outputs)[0]Intended uses & limitations
This is a LoRa adapter fine-tine for phi-2 and not a full fine-tune of the model. Additionally, I did not spend time updating parameters.
Training and evaluation data
The adapter was fine-tuned on a Google Colab A100 GPU using DPO and the distilabel-intel-orca-dpo-pairs. The full training notebook can be found here. Underneath, there are some configs for the adapter and the trainer.
peft_config = LoraConfig(
lora_alpha=16,
lora_dropout=0.5,
r=32,
target_modules=['k_proj', 'q_proj', 'v_proj', 'fc1', 'fc2'],
bias="none",
task_type="CAUSAL_LM",
)training_arguments = TrainingArguments(
output_dir=f"./{model_name}",
evaluation_strategy="steps",
do_eval=True,
optim="paged_adamw_8bit",
per_device_train_batch_size=2,
gradient_accumulation_steps=16,
per_device_eval_batch_size=2,
log_level="debug",
save_steps=20,
logging_steps=20,
learning_rate=1e-5,
eval_steps=20,
num_train_epochs=1, # Modified for tutorial purposes
max_steps=100,
warmup_steps=20,
lr_scheduler_type="linear",
)Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-05
- trainbatchsize: 2
- evalbatchsize: 2
- seed: 42
- gradientaccumulationsteps: 16
- totaltrainbatch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lrschedulertype: linear
- lrschedulerwarmup_steps: 20
- num_epochs: 1
Training results
Framework versions
- PEFT 0.7.1
- Transformers 4.37.1
- Pytorch 2.1.0+cu121
- Datasets 2.16.1
- Tokenizers 0.15.1