pszemraj/long-t5-tglobal-xl-qmsum-wip

long-t5-tglobal-xl-qmsum-wip

⚠️ warning - this is a work in progress ⚠️

<a href="https://colab.research.google.com/gist/pszemraj/ea0ac20dae4ad84bea4ea64543f84a85/long-t5-tglobal-xl-qmsum-wip.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> </a>

This model is a fine-tuned version of google/long-t5-tglobal-xl on the pszemraj/qmsum-cleaned dataset.

• —Refer to the dataset card for details but this model was trained with the task/prompt prefixes at the start of `input` which means that inference should be run in a similar fashion.
• —an example of how to run inference is in the Colab notebook linked above.

It achieves the following results on the evaluation set:

• —Loss: 2.0505
• —Rouge1: 35.3881
• —Rouge2: 11.509
• —Rougel: 23.1543
• —Rougelsum: 31.3295
• —Gen Len: 80.8

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• —learning_rate: 7e-05
• —trainbatchsize: 1
• —evalbatchsize: 1
• —seed: 2526
• —gradientaccumulationsteps: 8
• —totaltrainbatch_size: 8
• —optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• —lrschedulertype: cosine
• —lrschedulerwarmup_ratio: 0.03
• —num_epochs: 3.0

Training results