Nhoodie/enbed-dna-mutation-512bp

ENBED DNA Mutation Prediction (512bp, Checkpoint-200)

Fine-tuned variant of ENBED (ByT5-based DNA foundation model) for cross-domain mutation prediction across 9 domains of life, with emphasis on horizontal gene transfer (HGT) patterns.

Model Details

• —Base model: malusare/dna-base-512-pretrained (~581M params, 18 encoder / 6 decoder layers)
• —Fine-tuning task: Seq2seq DNA mutation prediction (parent → child sequence)
• —Training data: 4,143 mutation pairs across 9 domains (Bacteria, Archaea, Viruses, Retroviridae, Viridiplantae, Metazoa, Viroids, Satellites, Lichenes HGT)
• —Best checkpoint: Step 200 (eval loss 1.250)
• —Context length: 512 tokens (byte-level: 1 base = 1 token)
• —Hardware: Intel Hades Canyon NUC + GTX 1080 8GB eGPU

Training Configuration

• —Optimizer: Adafactor
• —Batch size: 1 (gradient accumulation: 16, effective batch = 16)
• —Learning rate: 1e-4
• —Precision: fp16 (model loaded in fp16, no AMP scaler)
• —Gradient checkpointing: enabled
• —Epochs: 5 (best at epoch ~0.8)

Eval Loss Progression

Model began overfitting after step 200. This checkpoint represents the best generalization point.

Intended Use

• —Cross-domain DNA mutation prediction
• —Horizontal gene transfer pattern analysis
• —Synthetic mutation data generation for underrepresented domains
• —Research baseline for DNA foundation model comparisons

Limitations

• —Overfitting: 581M params on 4K pairs → overfits quickly. Best used as a baseline or synthetic data generator.
• —Context: 512bp limit truncates longer sequences
• —Domains with few examples (Bacteria, Archaea) will have poor performance
• —HGT events (large insertions) are the hardest prediction task and likely near-random

Citation

If you use this model, please cite both the ENBED paper and this work:

@article{malusare2024enbed,
  title={ENBED: Ensemble Nucleotide Byte-level Encoder-Decoder for DNA Foundation Models},
  author={Malusare, Aditya and Lanman, Nadia and Aggarwal, Vaneet},
  journal={Bioinformatics Advances},
  year={2024}
}

License

MIT (following base model license)