Modelpublic

kanishka/opt-babylm1_no_non_preemption_laugh-1x-lemmatized_seed-1024_5e-6

sourceHugging Faceupdated 5mo agoView on Hugging Face

0likes16downloads

Model Card

opt-babylm1nononpreemptionlaugh-1x-lemmatizedseed-10245e-6

This model was trained from scratch on an unknown dataset. It achieves the following results on the evaluation set:

—Loss: 2.9128

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

—learning_rate: 5e-06
—trainbatchsize: 32
—evalbatchsize: 64
—seed: 1024
—gradientaccumulationsteps: 8
—totaltrainbatch_size: 256
—optimizer: Use OptimizerNames.ADAMWTORCHFUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
—lrschedulertype: linear
—lrschedulerwarmup_ratio: 0.05
—num_epochs: 20.0

Training results

Training Loss	Epoch	Step	Validation Loss
5.0219	0.4247	1000	5.1349
4.5239	0.8494	2000	4.5211
4.0882	1.2739	3000	4.0834
3.7572	1.6986	4000	3.7520
3.4828	2.1232	5000	3.4865
3.328	2.5479	6000	3.3416
3.2515	2.9726	7000	3.2442
3.1704	3.3971	8000	3.1889
3.1265	3.8218	9000	3.1435
3.0621	4.2463	10000	3.1081
3.0375	4.6710	11000	3.0791
2.9731	5.0956	12000	3.0571
2.9776	5.5203	13000	3.0396
2.9695	5.9449	14000	3.0190
2.9283	6.3695	15000	3.0127
2.9192	6.7942	16000	2.9953
2.878	7.2187	17000	2.9901
2.8776	7.6434	18000	2.9803
2.8283	8.0680	19000	2.9743
2.8449	8.4926	20000	2.9699
2.8483	8.9173	21000	2.9553
2.8127	9.3419	22000	2.9574
2.8166	9.7666	23000	2.9491
2.7812	10.1911	24000	2.9463
2.788	10.6158	25000	2.9415
2.7532	11.0403	26000	2.9434
2.7636	11.4650	27000	2.9349
2.7672	11.8897	28000	2.9277
2.7384	12.3143	29000	2.9309
2.7438	12.7390	30000	2.9259
2.7195	13.1635	31000	2.9278
2.7245	13.5882	32000	2.9237
2.7164	14.0127	33000	2.9241
2.6977	14.4374	34000	2.9202
2.7108	14.8621	35000	2.9164
2.6867	15.2867	36000	2.9195
2.6848	15.7114	37000	2.9162
2.662	16.1359	38000	2.9189
2.6714	16.5606	39000	2.9150
2.6747	16.9853	40000	2.9120
2.6593	17.4098	41000	2.9149
2.6541	17.8345	42000	2.9130
2.6399	18.2591	43000	2.9150
2.6445	18.6838	44000	2.9132
2.6278	19.1083	45000	2.9137
2.6347	19.5330	46000	2.9135
2.6279	19.9577	47000	2.9128

Framework versions

—Transformers 4.54.0
—Pytorch 2.10.0+cu128
—Datasets 3.2.0
—Tokenizers 0.21.4