zhangtaolab/PlantHelixSeek

PlantHelixSeek

A Hybrid Linear Attention-Transformer Mixture-of-Experts (MoE) DNA Language Model for Plants

Single-nucleotide resolution · Bidirectional MLM · 8,192 bp context · 470M params

PlantHelixSeek is a plant DNA foundation model pretrained with masked language modeling (MLM) at single-nucleotide resolution. It is the base model for all PlantHelixSeek fine-tuned checkpoints (sequence classification, token classification, gene structure prediction, CRE prediction).

Model Details

Architecture

HelixSeek interleaves three complementary layer types for efficient long-context bidirectional DNA modeling:

• —Transformer layers — full attention for precise positional reasoning
• —HelixSeek-Delta (KDA) — Delta Attention: linear attention with gated delta rule for O(n) long-range modeling
• —HelixSeek-MLA — Multi-head Latent Attention for KV-cache compression

The FFN uses a sparse Mixture-of-Experts with 8 experts (top-1 routing + 1 shared expert), achieving high capacity at constant inference cost.

Downstream Models

All downstream checkpoints are fine-tuned from this base:

Sequence Classification (binary)

• —PlantHelixSeek-core_promoter — Core promoter identification
• —PlantHelixSeek-lncRNA — lncRNA classification
• —PlantHelixSeek-open_chromatin — Open chromatin region
• —PlantHelixSeek-sequence_conservation — Sequence conservation
• —PlantHelixSeek-H3K27ac / -H3K27me3 / -H3K4me3 — Histone marks
• —PlantHelixSeek-translation_onoff — Translation on/off

Regression

• —PlantHelixSeek-promoter_strength — Promoter strength
• —PlantHelixSeek-terminator_strength — Terminator strength
• —PlantHelixSeek-translation_absolute — Translation level

Paper Downstream Tasks

• —PlantHelixSeek-Anno — Gene structure prediction (17 BILOU labels, token classification)
• —PlantHelixSeek-CRE — Cis-regulatory element prediction

Usage

from transformers import AutoModelForMaskedLM, AutoTokenizer
import torch

model = AutoModelForMaskedLM.from_pretrained("zhangtaolab/PlantHelixSeek", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("zhangtaolab/PlantHelixSeek", trust_remote_code=True)

# Mask recovery
seq = "ATCGGCTAAGCCT<mask>GTCAGTTACCGATCG"
inputs = tokenizer(seq, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits

Always pass trust_remote_code=True — the HelixSeek architecture is registered via HuggingFace auto_map.

License

CC-BY-NC 4.0 (research use; contact authors for commercial licensing)