Modelpublic

kanishka/smolm-autoreg-bpe-babylm-aann-counterfactual-anan-1e-4

sourceHugging Faceupdated 3y agoView on Hugging Face

0likes35downloads

Model Card

smolm-autoreg-bpe-babylm-aann-counterfactual-anan-1e-4

This model is a fine-tuned version of models/smolm-autoreg-bpe-babylm-aann-counterfactual-anan-1e-4/config.json on an unknown dataset. It achieves the following results on the evaluation set:

—Loss: 3.1756
—Accuracy: 0.4274

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

—learning_rate: 0.0001
—trainbatchsize: 32
—evalbatchsize: 128
—seed: 42
—optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
—lrschedulertype: linear
—lrschedulerwarmup_steps: 32000
—num_epochs: 20.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
3.9239	1.0	18353	3.9992	0.3336
3.3918	2.0	36706	3.5081	0.3842
3.201	3.0	55059	3.3534	0.4017
3.1032	4.0	73412	3.2792	0.4094
3.0303	5.0	91765	3.2313	0.4146
2.9788	6.0	110118	3.2123	0.4176
2.9353	7.0	128471	3.1918	0.4199
2.8982	8.0	146824	3.1838	0.4220
2.868	9.0	165177	3.1760	0.4230
2.8384	10.0	183530	3.1689	0.4237
2.8118	11.0	201883	3.1658	0.4250
2.7971	12.0	220236	3.1678	0.4250
2.7705	13.0	238589	3.1651	0.4258
2.7438	14.0	256942	3.1691	0.4260
2.73	15.0	275295	3.1655	0.4264
2.712	16.0	293648	3.1646	0.4269
2.6921	17.0	312001	3.1692	0.4271
2.6711	18.0	330354	3.1688	0.4273
2.657	19.0	348707	3.1738	0.4274
2.6387	20.0	367060	3.1756	0.4274

Framework versions

—Transformers 4.34.0
—Pytorch 2.0.1+cu117
—Datasets 2.12.0
—Tokenizers 0.14.1