chaoweihuang/FactAlign-LLaMA-3-8B

FactAlign-LLaMA-3-8B

This model is aligned with our FactAlign framework for improved long-form factuality, from meta-llama/Meta-Llama-3-8B-Instruct.

For more information, please refer to our paper: FactAlign: Long-form Factuality Alignment of Large Language Models.

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the trl-lib/kto-mix-14k and the chaoweihuang/lf-response-llama3-f11000.8-fg0.5 datasets. It achieves the following results on the evaluation set:

• —Loss: 0.4110
• —Rewards/chosen: 1.7360
• —Logps/chosen: -336.0412
• —Rewards/rejected: -2.2628
• —Logps/rejected: -406.1173
• —Rewards/margins: 3.9987
• —Kl: 0.0141
• —Fg Rewards/chosen Sum: -1.5560
• —Fg Logps/policy Chosen: -6.7332
• —Fg Logps/reference Chosen: -6.0419
• —Count/fg Chosen: 30.1832
• —Fg Rewards/rejected Sum: -0.9033
• —Fg Logps/policy Rejected: -8.6269
• —Fg Logps/reference Rejected: -7.5807
• —Count/fg Rejected: 6.9239
• —Fg Logps/policy Kl: -14.7946
• —Fg Logps/reference Kl: -11.4736
• —Fg Kl: nan
• —Fg Loss: 0.7625

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• —learning_rate: 5e-07
• —trainbatchsize: 1
• —evalbatchsize: 1
• —seed: 42
• —distributed_type: multi-GPU
• —num_devices: 4
• —gradientaccumulationsteps: 4
• —totaltrainbatch_size: 16
• —totalevalbatch_size: 4
• —optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• —lrschedulertype: linear
• —lrschedulerwarmup_ratio: 0.1
• —num_epochs: 1.0

Training results

Framework versions

• —Transformers 4.41.1
• —Pytorch 2.3.0+cu121
• —Datasets 2.20.0
• —Tokenizers 0.19.1