MoritzLaurer/ModernBERT-large-zeroshot-v2.0

ModernBERT-base-zeroshot-v2.0

Model description

This model is answerdotai/ModernBERT-large fine-tuned on the same dataset mix as the zeroshot-v2.0 models in the Zeroshot Classifiers Collection.

General takeaways:

• —The model is very fast and memory efficient. It's multiple times faster and consumes multiple times less memory than DeBERTav3. The memory efficiency enables larger batch sizes. I got a ~2x speed increase by enabling bf16 (instead of fp16).
• —It performs slightly worse then DeBERTav3 on average on the tasks tested below.
• —I'm in the process of preparing a newer version trained on better synthetic data to make full use of the 8k context window and to update the training mix of the older zeroshot-v2.0 models.

Training results

Training hyperparameters

The following hyperparameters were used during training:

• —learning_rate: 9e-06
• —trainbatchsize: 16
• —evalbatchsize: 32
• —seed: 42
• —gradientaccumulationsteps: 2
• —totaltrainbatch_size: 32
• —optimizer: Use adamwtorch with betas=(0.9,0.999) and epsilon=1e-08 and optimizerargs=No additional optimizer arguments
• —lrschedulertype: linear
• —lrschedulerwarmup_ratio: 0.06
• —num_epochs: 2

Framework versions

• —Transformers 4.48.0.dev0
• —Pytorch 2.5.1+cu124
• —Datasets 3.2.0
• —Tokenizers 0.21.0