orionweller/mmBERT-pretrain-p3-others

mmBERT Pre-training Data P3 Phase 1 of 3: Diverse multilingual pre-training data mixture (trained for 2.3T tokens) used to train the mmBERT model suite. NOTE: this is only P3 of the pre-training data due to HF limits, you need to download and combine all three into one folder This dataset contains the pre-training phase data used to train all mmBERT encoder models. The data is provided in MDS format ready for use with Composer and the ModernBERT training repository.… See the full description on the dataset page: https://huggingface.co/datasets/orionweller/mmBERT-pretrain-p3-others.