kaizerBox/RoFormer-summarization

1---2tags:3- generated_from_trainer4datasets:5- xsum6model-index:7- name: RoFormer-summarization8  results: []9---10 11<!-- This model card has been generated automatically according to the information the Trainer had access to. You12should probably proofread and complete it, then remove this comment. -->13 14# RoFormer-summarization15 16This model is a fine-tuned version of [](https://huggingface.co/) on the xsum dataset.17It achieves the following results on the evaluation set:18- Loss: 3.876319 20## Model description21 22More information needed23 24## Intended uses & limitations25 26More information needed27 28## Training and evaluation data29 30More information needed31 32## Training procedure33 34### Training hyperparameters35 36The following hyperparameters were used during training:37- learning_rate: 0.00138- train_batch_size: 839- eval_batch_size: 840- seed: 4241- gradient_accumulation_steps: 442- total_train_batch_size: 3243- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-0844- lr_scheduler_type: cosine45- lr_scheduler_warmup_steps: 10046- num_epochs: 347- mixed_precision_training: Native AMP48 49### Training results50 51| Training Loss | Epoch | Step  | Validation Loss |52|:-------------:|:-----:|:-----:|:---------------:|53| 4.5465        | 1.0   | 5762  | 4.0642          |54| 3.9616        | 2.0   | 11525 | 3.9113          |55| 3.8473        | 3.0   | 17286 | 3.8763          |56 57 58### Framework versions59 60- Transformers 4.35.261- Pytorch 2.1.0+cu11862- Datasets 2.15.063- Tokenizers 0.15.064