Junrulu/Llama-3-8B-Instruct-Iterative-SamPO

Model Card for Llama-3-8B-Instruct-Iterative-SamPO

This repository provides a fine-tuned version of Llama-3-8B-Instruct, using our proposed SamPO algorithm: Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence. We obey all licenses mentioned in llama3's work.

Performance

Evaluation Details

Five conditional benchmarks, using lm-evaluation-harness:

• —GSM8K: 8-shot, report strict match
• —IFEval: 3-shot, report instruction-level strict accuracy
• —PiQA: 3-shot, report accuracy
• —MMLU: 0-shot, report normalized accuracy
• —TruthfulQA: 3-shot, report accuracy of single-true mc1 setting

One open-ended benchmark, using official alpaca_eval:

• —AlpacaEval2: win rate (%) judged by GPT-4-turbo between the model's outputs vs. the GPT-4-turbo's response
• —LC AlpacaEval2: length-debiased win rate (%) of AlpacaEval2
• —Length in Tokens: the average output length of AlpacaEval2, calculated in tokens with Llama3's tokenizer

Input Format

The model is trained to use the following format:

<|start_header_id|>user<|end_header_id|>

{PROMPT}<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>

{Response}

Training hyperparameters

The following hyperparameters were used during DPO/SamPO training:

• —DPO beta: 0.1
• —learning_rate: 4e-7
• —totaltrainbatch_size: 128
• —optimizer: AdamW with beta1 0.9, beta2 0.999 and epsilon 1e-8
• —lrschedulertype: linear
• —lrschedulerwarmup_ratio: 0.1
• —Weight Decay: 0.0
• —num_epochs: 3.0
• —Specifically add above input format over training samples