phonemetransformers/babylm-subwords-2-gpt2_lm-model

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

babylm-subwords-2-gpt2_lm-model

This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset. It achieves the following results on the evaluation set:

• —Loss: 3.6434
• —Model Preparation Time: 0.0024
• —Perplexity: 38.2216
• —Bpc: 5.2563
• —Babyslm Test Lexical: 0.7310
• —Babyslm Test Syntactic: 0.7454
• —Blimp Supplement: 0.5693
• —Blimp Filtered: 0.7115
• —Blimp Supplement Hypernym: 0.5012
• —Blimp Supplement Qa Congruence Easy: 0.6406
• —Blimp Supplement Qa Congruence Tricky: 0.2667
• —Blimp Supplement Subject Aux Inversion: 0.7631
• —Blimp Supplement Turn Taking: 0.675
• —Blimp Adjunct Island Filtered: 0.7134
• —Blimp Anaphor Gender Agreement Filtered: 0.9248
• —Blimp Anaphor Number Agreement Filtered: 0.9506
• —Blimp Animate Subject Passive Filtered: 0.7419
• —Blimp Animate Subject Trans Filtered: 0.8678
• —Blimp Causative Filtered: 0.6932
• —Blimp Complex Np Island Filtered: 0.4444
• —Blimp Coordinate Structure Constraint Complex Left Branch Filtered: 0.4205
• —Blimp Coordinate Structure Constraint Object Extraction Filtered: 0.6133
• —Blimp Determiner Noun Agreement 1 Filtered: 0.9763
• —Blimp Determiner Noun Agreement 2 Filtered: 0.9409
• —Blimp Determiner Noun Agreement Irregular 1 Filtered: 0.8972
• —Blimp Determiner Noun Agreement Irregular 2 Filtered: 0.8890
• —Blimp Determiner Noun Agreement With Adj 2 Filtered: 0.9001
• —Blimp Determiner Noun Agreement With Adj Irregular 1 Filtered: 0.8524
• —Blimp Determiner Noun Agreement With Adj Irregular 2 Filtered: 0.8298
• —Blimp Determiner Noun Agreement With Adjective 1 Filtered: 0.9539
• —Blimp Distractor Agreement Relational Noun Filtered: 0.6294
• —Blimp Distractor Agreement Relative Clause Filtered: 0.5465
• —Blimp Drop Argument Filtered: 0.7043
• —Blimp Ellipsis N Bar 1 Filtered: 0.6185
• —Blimp Ellipsis N Bar 2 Filtered: 0.8261
• —Blimp Existential There Object Raising Filtered: 0.6798
• —Blimp Existential There Quantifiers 1 Filtered: 0.9548
• —Blimp Existential There Quantifiers 2 Filtered: 0.4061
• —Blimp Existential There Subject Raising Filtered: 0.8333
• —Blimp Expletive It Object Raising Filtered: 0.7628
• —Blimp Inchoative Filtered: 0.5860
• —Blimp Intransitive Filtered: 0.7339
• —Blimp Irregular Past Participle Adjectives Filtered: 0.6681
• —Blimp Irregular Past Participle Verbs Filtered: 0.7495
• —Blimp Irregular Plural Subject Verb Agreement 1 Filtered: 0.8433
• —Blimp Irregular Plural Subject Verb Agreement 2 Filtered: 0.8430
• —Blimp Left Branch Island Echo Question Filtered: 0.5977
• —Blimp Left Branch Island Simple Question Filtered: 0.5184
• —Blimp Matrix Question Npi Licensor Present Filtered: 0.2831
• —Blimp Npi Present 1 Filtered: 0.4345
• —Blimp Npi Present 2 Filtered: 0.5317
• —Blimp Only Npi Licensor Present Filtered: 0.7812
• —Blimp Only Npi Scope Filtered: 0.6452
• —Blimp Passive 1 Filtered: 0.8690
• —Blimp Passive 2 Filtered: 0.8339
• —Blimp Principle A C Command Filtered: 0.6279
• —Blimp Principle A Case 1 Filtered: 0.9978
• —Blimp Principle A Case 2 Filtered: 0.8678
• —Blimp Principle A Domain 1 Filtered: 0.9650
• —Blimp Principle A Domain 2 Filtered: 0.6066
• —Blimp Principle A Domain 3 Filtered: 0.5643
• —Blimp Principle A Reconstruction Filtered: 0.3630
• —Blimp Regular Plural Subject Verb Agreement 1 Filtered: 0.8337
• —Blimp Regular Plural Subject Verb Agreement 2 Filtered: 0.8508
• —Blimp Sentential Negation Npi Licensor Present Filtered: 0.9456
• —Blimp Sentential Negation Npi Scope Filtered: 0.4455
• —Blimp Sentential Subject Island Filtered: 0.4485
• —Blimp Superlative Quantifiers 1 Filtered: 0.6313
• —Blimp Superlative Quantifiers 2 Filtered: 0.7373
• —Blimp Tough Vs Raising 1 Filtered: 0.3956
• —Blimp Tough Vs Raising 2 Filtered: 0.7478
• —Blimp Transitive Filtered: 0.7039
• —Blimp Wh Island Filtered: 0.7042
• —Blimp Wh Questions Object Gap Filtered: 0.7055
• —Blimp Wh Questions Subject Gap Filtered: 0.8842
• —Blimp Wh Questions Subject Gap Long Distance Filtered: 0.9090
• —Blimp Wh Vs That No Gap Filtered: 0.9477
• —Blimp Wh Vs That No Gap Long Distance Filtered: 0.9566
• —Blimp Wh Vs That With Gap Filtered: 0.2622
• —Blimp Wh Vs That With Gap Long Distance Filtered: 0.0758

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• —learning_rate: 0.001
• —trainbatchsize: 32
• —evalbatchsize: 32
• —seed: 42
• —optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• —lrschedulertype: linear
• —lrschedulerwarmup_steps: 90000
• —training_steps: 400000

Training results

Framework versions

• —Transformers 4.44.2
• —Pytorch 2.4.0+cu118
• —Datasets 2.18.0
• —Tokenizers 0.19.1