MichelNivard/proteinLM-mixed-pretraining-v1

Pretraining mix for Protein language models

In order to construct a solid pre-training data mixture for protein language models we sample a mix of proteins from 3 sources:

1. 1.MG_Prot50 (https://huggingface.co/datasets/tattabio/OMG_prot50): meta-genomic proteins created by clustering the Open MetaGenomic dataset (OMG) at 50% sequence identity.
2. 2.UniRef50: UniProt proteins from across all species clustered to 50% sequences identity, downloaded form UniProt on 31th of March 2025
3. 3.UniRef90-mammals: UniProt proteins proteins clustered to 50% sequences identity from mammalian species. downloaded form UniProt on 31th of March 2025

This dataset consists of just shy of 40 billion amino-acids across 150 million proteins, that are then sorted across 10 files. Filesizes (in # amino-acids):

The file sizes are between 22 and 10 million proteins. From file 1 to 10 the proportion meta-genomic proteins goes down, and the proportion UniRef50 proteins goes up. In the last 3 files UniRef90 sequences of mammals are mixed in to ensure oversampling of proteins that are evolutionarially similar to human proteins.

The proteins are shuffled to ensure there is no clumping of protein that are similar (either in terms of species of structure) within batches to ensure smooth learning.

The data is designed to be used in serquence file "train_1.tsv" then file two etc etc. This ensures each file has more UniProt proteins and the final 3 files have increasing portions of mammalian proteins in them.

The data-mix is designed for protein language models that are to be finetuned to prediction/structure/interaction tasks that relate to human or other mammal proteins as we specifically mix in more, but less diverse, mammalian proteins into the training mix in the last 3 files.