AI4PD/ProtGPT3-MSA

1---2library_name: transformers3tags:4- biology5- protein-language-model6- protein-generation7- msa8- multiple-sequence-alignment9- few-shot-prompting10- homolog-conditioned-generation11- causal-lm12- mixture-of-experts13- transformers14---15 16# Model Card for ProtGPT3-MSA17 18 19## Model Description20 21ProtGPT3-MSA is a multiple-sequence, homolog-conditioned autoregressive protein language model. It is part of the [ProtGPT3 family](https://huggingface.co/collections/AI4PD/protgpt3-family), an open-source suite of promptable and aligned protein language models for protein sequence generation.22 23Unlike the single-sequence ProtGPT3 models, ProtGPT3-MSA can be prompted with sets of homologous protein sequences, enabling few-shot, family-conditioned protein generation without task-specific fine-tuning. At inference, users can provide homologous protein sequences as context and generate additional family-consistent sequences.24 25ProtGPT3-MSA was trained to autoregressively predict sets of 16 concatenated protein sequences, separated by a special token `<s>` (i.e., marking the protein boundaries). Therefore, at inference, the model should be prompted with at most 15 concatenated protein sequences.26 27- For more details on how to use  ProtGPT3-MSA check out our [colab](https://colab.research.google.com/drive/1HZFLUkRIhjUJdbQyvJC8ftio_ZHNL7kI?usp=sharing#scrollTo=zwWWcxwkPm6c).28 29- For a quick usage of the model for generating new sequences by prompting it with a fasta file of homologous sequences check out [ProtGPT3-MSA API](https://huggingface.co/spaces/AI4PD/ProtGPT3-MSA). 30 31 32### Model Modalities 331. **Aligned vs unaligned mode**:ProtGPT3-MSA has been trained to process concatenated sets of homologs in both "aligned" (i.e., the homologs are passed aligned with gap tokens) and "unaligned" mode via special `<gap>` and `<no_gap>` tokens, which should be placed at the start of the concatenated protein sequences to select the modality.34 352. **N-to-C vs C-to-N**:ProtGPT3-MSA has been trained to process concatenated homologs in both N-to-C and C-to-N directions, via two special "directional" tokens, "1" for N-to-C and "2" for C-to-N which which should be placed at the start of the concatenated protein sequences (i.e., before the gap token) to select the direction.36 37We provide some examples below.38 39## Uses40 41## How to Get Started with the Model42 43Install dependencies:44 45```bash46pip install transformers accelerate torch47```48 49Load the model and tokenizer:50 51```python52import torch53from transformers import AutoTokenizer, AutoModelForCausalLM54import random55import re56 57# ---- Intialise useful methods to prompt ProtGPT3-MSA ----58def process_style(seq: str, gap: bool):59    """Remove gaps, uppercase insertions, drop X."""60    if gap:61        # keep gaps62        return re.sub(r"[X]", "", seq.upper())63    else:64        # remove gaps65        return re.sub(r"[X]", "", seq.replace("-", "").upper())66 67def build_prompt(68    sequences: list, 69    gap: bool = False,70    direction: str ="1"71) -> str:72    """Build prompt for ProtGPT3-MSA73    Args:74      sequences: list of up to 15 homologous protein sequences (i.e., each entry in the list should be a homolog)75      gap: if True, process sequence in the aligned mode, so the homologs in sequences should be aligned (i.e., same length with gap tokens) if False sequences can be unaligned.76      direction: direction in which sequences should be processed/generated, "1": N-to-C, pass "2" to generate homologs in reversed C-to-N direction, importantly sequences should also be reversed if direction="2"  77    """78    assert len(sequences) <= 15, "The model cannot be prompted with more than 15 sequences (i.e., reduce the number of sequences to 15 or less)"79 80    # randomise order of sequences81    random.shuffle(sequences)82 83    if gap:84        gap_token = "<gap>"85        assert all(len(s) == len(sequences[0]) for s in sequences), "Sequences in the prompt have different len(), but should be aligned, either align them or use no_gap mode"86    else:87        gap_token = "<no_gap>"88 89    tokens: List[str] = ["<|bos|>", direction, gap_token]90    for seq in sequences:91        # add separator token between sequences92        tokens.append("<s>")93        tokens.extend(list(process_style(seq,gap=gap)))94 95    # Match train-time separator before continuation96    tokens.append("<s>")97    return " ".join(tokens)98## --------------------------------------99 100model_id = "AI4PD/ProtGPT3-MSA"101 102# Load tokenizer for generation103tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True,add_bos_token=False, add_eos_token=False, padding_side="left") # BOS token manually added in build_prompt104 105model = AutoModelForCausalLM.from_pretrained(106    model_id,107    torch_dtype=torch.bfloat16,108    device_map="auto",109    trust_remote_code=True,110)111 112model.eval()113```114 115### Few-shot generation with unaligned homologs116 117Use the `<no_gap>` modality token for unaligned sequences. Separate homologous sequences with the `<s>` separator token.118 119```python120import torch121 122 123homologs = [124    "MKTAYIAKQRQISFVKSHFSRQDILD",125    "MKTVYIAKQRQISFVKSHFSRQDILD",126    "MKTAYIAKQRQINNVKSHFSRQNILD",127    # Add up to 15 homologous protein sequences128]129 130prompt = build_prompt(sequences=homologs)131 132inputs = tokenizer(prompt, return_tensors="pt", padding=True).to(model.device)133 134with torch.no_grad():135    output_ids = model.generate(136        inputs["input_ids"],137        max_new_tokens=512, # CHANGE to desire length (i.e., protein length times n. of generated homologs sequentially)138        do_sample=True,139        temperature=0.8,140        top_p=0.9,141        eos_token_id=tokenizer.eos_token_id,142        pad_token_id=tokenizer.pad_token_id,143        num_return_sequences=20, # set to desired number of protein sequences to be generated in parallel144    )145 146generated = tokenizer.decode(output_ids[0], skip_special_tokens=True)147# split sequences generated sequentially148segments = generated.split("<s>")149# print each sequence150for s in segments:151  print(s.replace(" ",""),"\n")152```153 154### Few-shot generation with aligned homologs155 156Use the `<gap>` modality token for aligned sequences. Gap characters may be included in the prompted sequences.157 158```python159import torch160 161# must have the same length and be aligned162aligned_homologs = [163    "MKTAYIAKQRQI--SFVKSHFSRQDILD",164    "MKTVYIAKQRQI--SFVKSHFSRQDILD",165    "MKTAYIAKQRQINNSFVKSHFSRQNILD",166]167 168prompt = build_prompt(sequences=aligned_homologs, gap=True)169 170inputs = tokenizer(prompt, return_tensors="pt", padding=True).to(model.device)171 172with torch.no_grad():173    output_ids = model.generate(174        inputs["input_ids"],175        max_new_tokens=512, # CHANGE to desire length (i.e., protein length times n. of generated homologs sequentially)176        do_sample=True,177        temperature=0.8,178        top_p=0.9,179        eos_token_id=tokenizer.eos_token_id,180        pad_token_id=tokenizer.eos_token_id,181        num_return_sequences=20, # set to desired number of protein sequences to be generated in parallel 182    )183 184 185generated = tokenizer.decode(output_ids[0], skip_special_tokens=True)186# split sequences generated sequentially187segments = generated.split("<s>")188# print each sequence189for s in segments:190  print(s.replace(" ",""),"\n")191```192 193### Notes on prompting194 195- Use `<no_gap>` for unaligned homologous sequences.196- Use `<gap>` for aligned MSA-style inputs containing gap characters.197- Separate protein sequences with `<s>`.198- Provide up to 15 homologous sequences as context.199- Sampling parameters such as `temperature` and `top_p` can affect sequence quality, diversity, and family consistency.200- Generated sequences should be validated before experimental use.201- Change `max_new_tokens` in `generate()` to control the number of protein generated sequentially (i.e., you are passing 15 homologs as prompt, this should roughly equal the length of a single protein).202- - Use `num_return_sequences` in `generate()` to control the number of protein generated in parallel given the same prompt.203 204### Out-of-Scope Use205 206The model should not be used as the sole basis for experimental, clinical, environmental, or safety-critical decisions. Generated sequences require downstream computational and experimental validation. The model is not guaranteed to generate functional, soluble, safe, synthesizable, or experimentally successful proteins.207 208The model should not be used for irresponsible or harmful biological design applications.209 210## Bias, Risks, and Limitations211 212ProtGPT3-MSA learns from public protein sequence and MSA datasets and may reproduce biases present in those datasets. The model depends on the quality, relevance, and diversity of the homologous sequences provided in the prompt. Poor, unrelated, noisy, contaminated, or incorrectly aligned prompts may reduce generation quality.213 214Generated sequences may be nonfunctional, unstable, insoluble, repetitive, low-complexity, or biologically implausible. As with other generative protein models, ProtGPT3-MSA may present dual-use risks if applied irresponsibly.215 216### Recommendations217 218Users should provide high-quality homologous protein sequences and validate generated sequences with appropriate downstream computational and experimental methods. For family-conditioned generation, users should carefully curate prompts and assess generated sequences using task-relevant criteria such as sequence identity, structural confidence, family-level consistency, solubility, and functional plausibility.219 220## Training Details221 222### Training Data223 224ProtGPT3-MSA was trained on approximately 8.5M MSAs from the OpenProteinSet Uniclust30 dataset. From each MSA, 16 sequences were sampled without replacement and concatenated in random order. This process was repeated 15 times for each MSA, resulting in approximately 560B training tokens.225 226 227## Technical Specifications228 229### Model Architecture and Objective230 231ProtGPT3-MSA is a decoder-only autoregressive protein language model using a Mixtral-style sparse Mixture-of-Experts architecture. It was trained to model concatenated sets of related protein sequences, enabling homolog-conditioned generation through prompting.232 233The model processes up to 16 concatenated protein sequences and supports both aligned and unaligned modalities. During inference, users may provide up to 15 homologous sequences and generate an additional sequence conditioned on the prompt.234 235 236## Citation237 238**BibTeX:**239 240```bibtex241@article{garibbo2026protgpt3,242  title={ProtGPT3: an Open-source family of Promptable and Aligned Protein Language Models},243  author={Garibbo, Michele and Boxo Corominas, Gerard and Stocco, Filippo and Illanes Vicioso, Ramiro and Middendorf, Lasse and Ferruz, Noelia},244  journal={bioRxiv},245  pages={2026--06},246  year={2026},247  publisher={Cold Spring Harbor Laboratory}248}249```250 251 252## More Information253 254All models and code are released through the Hugging Face ecosystem and accompanying code repository.255