vesteinn/gpt2-dna

DNA Language Model (Char-level, Human-only)

This model is a character-level GPT-style language model trained exclusively on human DNA. It uses a custom tokenizer with a vocabulary of A, C, G, T, and a special end-of-text token, trained to predict the next base in 1024-base sequences.

🧬 Model Summary

• —Objective: Next-token prediction over human genomic sequences
• —Tokenization: Character-level (A, C, G, T)
• —Training data: simecek/Human\_DNA\_v0
• —Sequence length: 1024 tokens
• —Final Validation Loss: 1.0299 nats/token
• —Final Validation Accuracy: 53.24%

This model outperforms classical compressors like GeCo on human DNA entropy, achieving \~1.486 bits per base.

🔧 Tokenizer

The tokenizer is a minimal GPT-2-style vocabulary:

{
  "<|endoftext|>": 0,
  "A": 1,
  "C": 2,
  "G": 3,
  "T": 4
}

• —Implemented via GPT2TokenizerFast
• —Merges file is empty (no BPE applied)

📊 Dataset Preprocessing

• —Original dataset is cleaned to keep only A, C, G, T
• —Sequences are chunked into segments of length 1024
• —Very short chunks (<200bp) are discarded
• —A 10% split validation is made from the training set.

🚀 Intended Uses

This model can be used for:

• —DNA sequence generation
• —Genomic representation learning
• —Predictive modeling for base-level structure
• —Downstream fine-tuning for biological classification tasks

Limitations

• —Trained only on human genome; not suitable for other species
• —No reverse-complement modeling
• —No masked language modeling objective

🏋️ Training Details

Hyperparameters

• —learning\_rate: 0.0003
• —train\batch\size: 64
• —eval\batch\size: 8
• —total\train\batch\_size: 256 (across 4 GPUs)
• —total\eval\batch\_size: 32
• —optimizer: AdamW (betas=(0.9, 0.999), epsilon=1e-08)
• —lr\_scheduler: Linear with 1000 warmup steps
• —epochs: 10.0
• —mixed\_precision: Native AMP

Hardware

• —Multi-GPU training (4 devices)
• —Transformers 4.52.0.dev0
• —PyTorch 2.3.0+cu121

📈 Training Results

🔗 References

• —Tokenizer inspired by GPT-2 minimal vocab
• —Dataset: simecek/Human\_DNA\_v0
• —Transformers: https://github.com/huggingface/transformers
• —PyTorch: https://pytorch.org/

📄 Citation

This model is part of ongoing research. A formal citation will be added when the associated paper is published.

If you use this model in academic work, please check back for updates.