Heralax/army-pretrain-1

The pretrained base of the Mistrillitary model.

Training hyperparameters

The following hyperparameters were used during training:

• —learning_rate: 2e-05
• —trainbatchsize: 2
• —evalbatchsize: 1
• —seed: 42
• —distributed_type: multi-GPU
• —num_devices: 6
• —gradientaccumulationsteps: 6
• —totaltrainbatch_size: 72
• —totalevalbatch_size: 6
• —optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• —lrschedulertype: cosine
• —lrschedulerwarmup_steps: 136
• —num_epochs: 11

Training results

Framework versions

• —Transformers 4.45.0.dev0
• —Pytorch 2.3.1+cu121
• —Datasets 2.21.0
• —Tokenizers 0.19.1