CoolFace
Modelpublic

devika-tiwari/gpt2_small_expandedbabyLM_200M_43

sourceHugging Faceupdated 1mo agoView on Hugging Face
0likes4.1kdownloads
Model Card

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

gpt2smallexpandedbabyLM200M43

This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset. It achieves the following results on the evaluation set:

  • —Loss: 2.6807

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • —learning_rate: 0.0001
  • —trainbatchsize: 256
  • —evalbatchsize: 256
  • —seed: 43
  • —optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • —lrschedulertype: linear
  • —lrschedulerwarmup_steps: 4000
  • —num_epochs: 20

Training results

Training LossEpochStepValidation Loss
3.30521.0101632.9740
3.13882.0203262.8510
3.05533.0304892.8000
3.01214.0406522.7724
2.98025.0508152.7551
2.94146.0609782.7402
2.92037.0711412.7318
2.90928.0813042.7239
2.88379.0914672.7151
2.877810.01016302.7111
2.860711.01117932.7086
2.843912.01219562.7022
2.836313.01321192.6984
2.815614.01422822.6943
2.803615.01524452.6909
2.797416.01626082.6877
2.781417.01727712.6863
2.774618.01829342.6837
2.758319.01930972.6816
2.751220.02032602.6807

Framework versions

  • —Transformers 4.30.2
  • —Pytorch 2.11.0+cu130
  • —Datasets 4.1.1
  • —Tokenizers 0.13.3