Modelpublic

kanishka/smolm-autoreg-bpe-counterfactual_babylm_300_anans_new-1e-3

sourceHugging Faceupdated 2y agoView on Hugging Face

0likes29downloads

Model Card

smolm-autoreg-bpe-counterfactualbabylm300anansnew-1e-3

This model was trained from scratch on the kanishka/counterfactualbabylm300anansnew dataset. It achieves the following results on the evaluation set:

—Loss: 3.3718
—Accuracy: 0.4142

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

—learning_rate: 0.001
—trainbatchsize: 32
—evalbatchsize: 64
—seed: 42
—optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
—lrschedulertype: linear
—lrschedulerwarmup_steps: 32000
—num_epochs: 20.0
—mixedprecisiontraining: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
3.6009	1.0	18595	3.7621	0.3598
3.3788	2.0	37190	3.5666	0.3811
3.2578	3.0	55785	3.4510	0.3934
3.1725	4.0	74380	3.4047	0.3989
3.12	5.0	92975	3.3603	0.4034
3.0784	6.0	111570	3.3602	0.4069
3.0408	7.0	130165	3.3407	0.4081
3.0111	8.0	148760	3.3320	0.4090
2.9809	9.0	167355	3.3122	0.4133
2.961	10.0	185950	3.3328	0.4115
2.9322	11.0	204545	3.3232	0.4127
2.912	12.0	223140	3.3236	0.4120
2.894	13.0	241735	3.3336	0.4126
2.8682	14.0	260330	3.3342	0.4138
2.8494	15.0	278925	3.3361	0.4145
2.8289	16.0	297520	3.3370	0.4150
2.8119	17.0	316115	3.3507	0.4139
2.7921	18.0	334710	3.3529	0.4143
2.7755	19.0	353305	3.3627	0.4145
2.761	20.0	371900	3.3718	0.4142

Framework versions

—Transformers 4.40.1
—Pytorch 2.1.0+cu121
—Datasets 2.16.1
—Tokenizers 0.19.1