Cornell-AGI/REFUEL-Llama-3-Armo-iter_1

This is a model released for our paper: Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF.

REFUEL-Llama-3-Armo-iter_1

This model is developed with REFUEL based on Meta-Llama-3-8B-Instruct with ArmoRM-Llama3-8B-v0.1 as the reward model and UltraInteract dataset. The training code is available at https://github.com/ZhaolinGao/REFUEL.

Evaluations

<table> <tr> <th rowspan="2">Method</th> <th rowspan="2">Dataset</th> <th colspan="6">Winrate at Turn</th> </tr> <tr> <th>h = 1</th> <th>h = 2</th> <th>h = 3</th> <th>h = 4</th> <th>H = 5</th> <th>avg</th> </tr> <tr> <td>Llama-3.1-70B-it</td> <td> N/A </td> <td>70.4</td> <td>66.4</td> <td>61.0</td> <td>53.0</td> <td>55.4</td> <td>61.24</td> </tr> <tr> <td><a href="https://huggingface.co/Cornell-AGI/REFUEL-Llama-3-Armo-iter1">REFUEL-Llama-3-Armo-iter1</a></td> <td><a href="https://huggingface.co/datasets/Cornell-AGI/REFUEL-Ultrainteract-Llama-3-Armo-iter1">REFUEL-Ultrainteract-Llama-3-Armo-iter1</a></td> <td>54.6</td> <td>53.6</td> <td>57.8</td> <td>56.2</td> <td>59.4</td> <td>56.32</td> </tr> <tr> <td><a href="https://huggingface.co/Cornell-AGI/REFUEL-Llama-3-Armo-iter2">REFUEL-Llama-3-Armo-iter2</a></td> <td><a href="https://huggingface.co/datasets/Cornell-AGI/REFUEL-Ultrainteract-Llama-3-Armo-iter2">REFUEL-Ultrainteract-Llama-3-Armo-iter2</a></td> <td>55.2</td> <td>53.4</td> <td>58.8</td> <td>57.2</td> <td>58.6</td> <td>56.64</td> </tr> </table>

Citation

Please cite our paper if you use this model in your own work:

@misc{gao2024regressingrelativefutureefficient,
      title={Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF}, 
      author={Zhaolin Gao and Wenhao Zhan and Jonathan D. Chang and Gokul Swamy and Kianté Brantley and Jason D. Lee and Wen Sun},
      year={2024},
      eprint={2410.04612},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2410.04612}, 
}