masterkristall/bert_distillation_tiny

1---2license: apache-2.03base_model: google/bert_uncased_L-2_H-128_A-24tags:5- generated_from_trainer6datasets:7- glue8metrics:9- accuracy10model-index:11- name: bert_distillation_tiny12  results:13  - task:14      name: Text Classification15      type: text-classification16    dataset:17      name: glue18      type: glue19      config: sst220      split: validation21      args: sst222    metrics:23    - name: Accuracy24      type: accuracy25      value: 0.825688073394495526---27 28<!-- This model card has been generated automatically according to the information the Trainer had access to. You29should probably proofread and complete it, then remove this comment. -->30 31# bert_distillation_tiny32 33This model is a fine-tuned version of [google/bert_uncased_L-2_H-128_A-2](https://huggingface.co/google/bert_uncased_L-2_H-128_A-2) on the glue dataset.34It achieves the following results on the evaluation set:35- Loss: 0.427436- Accuracy: 0.825737 38## Model description39 40More information needed41 42## Intended uses & limitations43 44More information needed45 46## Training and evaluation data47 48More information needed49 50## Training procedure51 52### Training hyperparameters53 54The following hyperparameters were used during training:55- learning_rate: 0.000156- train_batch_size: 12857- eval_batch_size: 12858- seed: 202359- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-0860- lr_scheduler_type: linear61- num_epochs: 762- mixed_precision_training: Native AMP63 64### Training results65 66| Training Loss | Epoch | Step | Validation Loss | Accuracy |67|:-------------:|:-----:|:----:|:---------------:|:--------:|68| 0.4139        | 1.0   | 527  | 0.4204          | 0.8096   |69| 0.27          | 2.0   | 1054 | 0.4274          | 0.8257   |70| 0.2226        | 3.0   | 1581 | 0.4899          | 0.8245   |71| 0.1931        | 4.0   | 2108 | 0.4961          | 0.8222   |72| 0.1732        | 5.0   | 2635 | 0.5302          | 0.8222   |73| 0.1608        | 6.0   | 3162 | 0.5393          | 0.8234   |74| 0.152         | 7.0   | 3689 | 0.5562          | 0.8177   |75 76 77### Framework versions78 79- Transformers 4.35.280- Pytorch 2.1.0+cu11881- Datasets 2.15.082- Tokenizers 0.15.083