igemugm/dnabert-nicotiana

🧬 DNABERT-2 Domain-Adaptive Pretraining (DAP) on Nicotiana tabacum

This model is a domain-adaptively pre-trained DNABERT-2 on Nicotiana tabacum DNA sequences. The purpose of this pretraining is to adapt the base DNABERT-2 model to capture plant-specific DNA patterns before fine-tuning on downstream tasks.

📘 Overview

• —Base model: `zhihan1996/DNABERT-2-117M`
• —Pretraining task: Masked Language Modeling (MLM)
• —Domain: Nicotiana tabacum DNA sequences
• —Tokenizer: DNABERT-2 tokenizer
• —Dataset size: ~1% of available Nicotiana sequences
• —Objective: Predict 15% randomly masked tokens in each sequence

⚙️ Training Details

• —Training epochs: 10
• —Early stopping: patience = 3 (monitoring validation loss)
• —Learning rate: 2e-5
• —Weight decay: 0.01
• —Batch size: 8 per device
• —Sequence length: max 1024 tokens
• —Padding: Dynamic, via DataCollatorForLanguageModeling

Training procedure:

1. 1.Tokenize DNA sequences with DNABERT-2 tokenizer
2. 2.Randomly mask 15% of tokens
3. 3.Train with MLM objective
4. 4.Apply early stopping based on validation loss

📊 Evaluation

Both training and validation losses gradually decreased, showing that the model learned patterns in Nicotiana tabacum DNA sequences without overfitting. Final training loss: ~5.55, validation loss: ~5.53. Final loss are relatively high due to the limited dataset (1% of total available data).

💡 Usage

This model can be used for masked language modeling tasks or as a pretraining checkpoint for downstream tasks like stress region prediction or other plant DNA sequence analyses.

from transformers import AutoTokenizer, BertModel
import torch

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("igemugm/dnabert-nicotiana", trust_remote_code=True)
model = BertModel.from_pretrained("igemugm/dnabert-nicotiana", trust_remote_code=True)

# Input DNA sequence
sequence = "ACGTAGCATCGGATCTATCTATCGACACTTGGTTATCGATCTACGAGCATCTCGTTAGC"
inputs = tokenizer(sequence, return_tensors="pt")

# Forward pass for hidden states
with torch.no_grad():
    outputs = model(**inputs)
    hidden_states = outputs.last_hidden_state  # [batch, seq_len, hidden_dim]

# CLS token embedding
cls_embedding = hidden_states[:, 0, :]  # [batch, hidden_dim]

# Mean pooling embedding
mean_embedding = hidden_states.mean(dim=1)  # [batch, hidden_dim]

print("CLS embedding shape:", cls_embedding.shape)
print("Mean embedding shape:", mean_embedding.shape)