Salesforce/LLaMA-3-8B-SFR-Iterative-DPO-R
78105
1---2license: llama33---4# Llama-3-8B-SFR-Iterative-DPO-R5 6## Introduction7We release a state-of-the-art instruct model of its class, **Llama-3-8B-SFR-Iterative-DPO-R**.8On all three widely-used instruct model benchmarks: **Alpaca-Eval-V2**, **MT-Bench**, **Chat-Arena-Hard**, our model outperforms all models of similar size (e.g., LLaMA-3-8B-it), most large open-sourced models (e.g., Mixtral-8x7B-it),9and strong proprietary models (e.g., GPT-3.5-turbo-0613). The model is trained with open-sourced datasets without any additional human-/GPT4-labeling.10 11## Model Releases12- [SFT model](https://huggingface.co/Salesforce/SFR-SFT-LLaMA-3-8B-R)13- [Reward model](https://huggingface.co/Salesforce/SFR-RM-LLaMA-3-8B-R)14- [RLHF model](https://huggingface.co/Salesforce/SFR-Iterative-DPO-LLaMA-3-8B-R)15 16 17## Training methods18We have developed a simple and efficient online RLHF recipe for LLM instruct training. Our recipe is DPO-based and thus much cheaper and simpler to train and tune compared to PPO-based approaches.19Unlike widely-used offline DPO, the online component of our approach effectively mitigates distribution shifts during policy optimization.20For a detailed exposition, please refer to our accompanying technical report.21 22 23## Chat Benchmarks24 25| **Model** | **Size** | **Method** | **LC Alpaca-Eval-V2** | **MT-Bench** | **Chat-Arena-Hard** |26|-------------------------|----------|-------------------|-----------------------|--------------|---------------------|27| **Small Open-Sourced Models** | | | | | |28| Gemma-7B-it | 7B | SFT | 10.4 | 6.38 | 7.5 |29| Zephyr-7B-beta | 7B | Vanilla DPO | 13.1 | 7.34 | - |30| Mistral-7B-v0.2-it | 7B | SFT | 17.1 | 7.51 | 12.6 |31| Open-Chat-0106 | 7B | SFT | 15.6 | 7.8 | - |32| Starling-7B-beta | 7B | PPO | 25.8 | 8.12 | 23.0 |33| LLaMA-3-8B-it | 8B | RS+DPO+PPO | 22.9 | 8.16 | 20.6 |34| **Ours** | | | | | |35| Ours (SFT baseline) | 8B | SFT | 10.2 | 7.69 | 5.6 |36| Ours (DPO baseline) | 8B | Vanilla DPO | 22.5 | 8.17 | 22.4 |37| Ours (Online RLHF) | 8B | Iterative DPO | **31.3** | **8.46** | **29.1** |38| **Large Open-Sourced Models** | | | | | |39| Vicuna-33b-v1.3 | 33B | SFT | 17.6 | 7.12 | 8.6 |40| Yi-34B-Chat | 34B | SFT | 27.2 | - | 23.1 |41| Mixtral-8x7B-it | 45B* | SFT | 23.7 | 8.30 | 23.4 |42| Tulu-2-DPO-70B | 70B | Vanilla DPO | 21.2 | 7.89 | 15.0 |43| LLaMA-3-70B-it | 70B | RS+DPO+PPO | 34.4 | 8.95 | 41.1 |44| Mixtral-8x22B-it | 141B* | SFT | 30.9 | 8.66 | 36.4 |45| **Proprietary Models** | | | | | |46| GPT-3.5-turbo-1106 | - | - | 19.3 | 8.35 | 18.9 |47| GPT-3.5-turbo-0613 | - | - | 22.7 | 8.39 | 24.8 |48| GPT-4-0613 | - | - | 30.2 | 9.18 | 37.9 |49| Claude-3-Opus | - | - | 40.5 | 9.00 | 60.4 |50| GPT-4 Turbo (04/09) | - | - | 55.0 | - | 82.6 |51 52 53## Academic Benchmarks54 55| **Model** | **Size** | **Method** | **GSM-8K** | **MMLU** | **HumanEval** | **TruthfulQA** | **ARC** | **MBPP** |56|----------------------------|----------|-----------------|------------|----------|---------------|----------------|---------|----------|57| LLaMA-3-8B-it | 8B | RS+DPO+PPO | 79.6 | 66.0 | 61.6 | 43.9 | 59.5 | 61.1 |58| Ours (SFT baseline) | 8B | SFT | 74.2 | 64.7 | 65.2 | 53.4 | 61.4 | 62.3 |59| Ours (DPO baseline) | 8B | Vanilla DPO | 79.8 | 64.5 | 63.4 | 61.8 | 65.2 | 60.3 |60| Ours (Iterative RLHF) | 8B | Iterative DPO | 80.7 | 65.3 | 64.6 | 60.4 | 64.3 | 60.8 |61 62 63## Usage64```python65from transformers import AutoModelForCausalLM, AutoTokenizer66 67device = "cuda" 68 69model = AutoModelForCausalLM.from_pretrained("Salesforce/Llama-3-8B-SFR-Iterative-DPO-R")70tokenizer = AutoTokenizer.from_pretrained("Salesforce/Llama-3-8B-SFR-Iterative-DPO-R")71 72messages = [73 {"role": "user", "content": "I'm trying to teach myself to have nicer handwriting. Can you help?"},74]75 76model_inputs = tokenizer.apply_chat_template(messages, return_tensors="pt")77 78model_inputs = model_inputs.to(device)79model.to(device)80 81output_tokens = model.generate(model_inputs, max_new_tokens=1024, do_sample=True)82model_outputs = tokenizer.batch_decode(output_tokens)83print(model_outputs[0])84```85 86 87## Limitations88Llama-3-8B-SFR-Iterative-DPO-R is a research model developed as part of our RLHF initiative at Salesforce. 89While safety and ethical considerations are integral to our alignment process, 90there remains the possibility that the model could generate offensive or unethical content, particularly under adversarial conditions. 91We are committed to continuous improvement in our models to minimize such risks and encourage responsible usage.92 93## Citation94Please cite our papers if you find our models are useful.95 96```bibtex97@misc{dong2024rlhf,98 title={RLHF Workflow: From Reward Modeling to Online RLHF}, 99 author={Hanze Dong* and Wei Xiong* and Bo Pang* and Haoxiang Wang* and Han Zhao and Yingbo Zhou and Nan Jiang and Doyen Sahoo and Caiming Xiong and Tong Zhang},100 year={2024},101 eprint={2405.07863},102 archivePrefix={arXiv},103 primaryClass={cs.LG}104}105 106@misc{xiong2024iterative,107 title={Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint}, 108 author={Wei Xiong and Hanze Dong and Chenlu Ye and Ziqi Wang and Han Zhong and Heng Ji and Nan Jiang and Tong Zhang},109 year={2024},110 eprint={2312.11456},111 archivePrefix={arXiv},112 primaryClass={cs.LG}113}114```115 116## Ethics disclaimer for Salesforce AI models, data, code117 118This release is for research purposes only in support of an academic119paper. Our models, datasets, and code are not specifically designed or120evaluated for all downstream purposes. We strongly recommend users121evaluate and address potential concerns related to accuracy, safety, and122fairness before deploying this model. We encourage users to consider the123common limitations of AI, comply with applicable laws, and leverage best124practices when selecting use cases, particularly for high-risk scenarios125where errors or misuse could significantly impact people’s lives, rights,126or safety. For further guidance on use cases, refer to our standard127[AUP](https://www.salesforce.com/content/dam/web/en_us/www/documents/legal/Agreements/policies/ExternalFacing_Services_Policy.pdf)128and [AI AUP](https://www.salesforce.com/content/dam/web/en_us/www/documents/legal/Agreements/policies/ai-acceptable-use-policy.pdf).