Modelpublic

AIPlans/tinyllama-1.1b-dpo-pku-saferlhf

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face

0likes28downloads

Model Card

tinyllama-1.1b-dpo-pku-saferlhf

This model is a fine-tuned version of TinyLlama/TinyLlama-1.1B-Chat-v1.0 on an unknown dataset. It achieves the following results on the evaluation set:

—Loss: 0.6742
—Rewards/chosen: 0.0508
—Rewards/rejected: 0.0098
—Rewards/accuracies: 0.6750
—Rewards/margins: 0.0410
—Logps/rejected: -205.4332
—Logps/chosen: -232.6254
—Logits/rejected: -2.5959
—Logits/chosen: -2.7163

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

—learning_rate: 5e-07
—trainbatchsize: 4
—evalbatchsize: 4
—seed: 42
—gradientaccumulationsteps: 4
—totaltrainbatch_size: 16
—optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
—lrschedulertype: cosine
—lrschedulerwarmup_ratio: 0.1
—num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6899	0.0960	200	0.6898	0.0058	-0.0017	0.5680	0.0074	-205.5475	-233.0753	-2.5954	-2.7156
0.6868	0.1920	400	0.6851	0.0253	0.0080	0.6300	0.0174	-205.4510	-232.8795	-2.5955	-2.7157
0.6741	0.2880	600	0.6806	0.0368	0.0096	0.6410	0.0272	-205.4345	-232.7647	-2.5956	-2.7159
0.6767	0.3840	800	0.6753	0.0499	0.0115	0.6670	0.0384	-205.4159	-232.6337	-2.5954	-2.7158
0.676	0.4800	1000	0.6766	0.0494	0.0133	0.6570	0.0361	-205.3982	-232.6393	-2.5960	-2.7163
0.6774	0.5760	1200	0.6742	0.0508	0.0098	0.6750	0.0410	-205.4332	-232.6254	-2.5959	-2.7163
0.6746	0.6719	1400	0.6737	0.0529	0.0107	0.6550	0.0422	-205.4236	-232.6038	-2.5958	-2.7162
0.6678	0.7679	1600	0.6743	0.0513	0.0104	0.6630	0.0409	-205.4272	-232.6203	-2.5960	-2.7164
0.6833	0.8639	1800	0.6739	0.0528	0.0108	0.6620	0.0420	-205.4227	-232.6052	-2.5959	-2.7163
0.6743	0.9599	2000	0.6741	0.0530	0.0115	0.6450	0.0415	-205.4160	-232.6033	-2.5958	-2.7163

Framework versions

—Transformers 4.44.2
—Pytorch 2.10.0+cu130
—Datasets 2.21.0
—Tokenizers 0.19.1