multimolecule/enformer

1---2datasets:3- multimolecule/encode4- multimolecule/fantom55- multimolecule/gtex6library_name: multimolecule7license: agpl-3.08pipeline: regulatory-track9pipeline_tag: other10tags:11- Biology12- DNA13- dna14widget:15- example_title: tumor protein p5316  pipeline_tag: regulatory-track17  sequence_type: DNA18  task: regulatory-track19  text: ACTCCCCTGCCCTCAACAAGATGTTTTGCCAACTGGCCAAGACCTGCCCTGTGCAGCTGTGGGTTGATTCCACACCCCCGCCCGGCACCCGCGTCCGCGCCATGGCCATCTACAAGCAGTCACAGCACATGACGGAGGTTGTGAGGCGCTGCCCCCACCATGAGCGCTGCTCAGATAGCGATGG20- example_title: BRCA1 DNA repair associated21  pipeline_tag: regulatory-track22  sequence_type: DNA23  task: regulatory-track24  text: TCATTGGAACAGAAAGAAATGGATTTATCTGCTCTTCGCGTTGAAGAAGTACAAAATGTCATTAATGCTATGCAGAAAATCTTAGAGTGTCCCATCTGG25- example_title: hemoglobin subunit beta26  pipeline_tag: regulatory-track27  sequence_type: DNA28  task: regulatory-track29  text: CATTTGCTTCTGACACAACTGTGTTCACTAGCAACCTCAAACAGACACCATGGTGCATCTGACTCCTGAGGAGAAGTCTGCCGTTACTGCCCTGTGGGGCAAGGTGAACGTGGATGAAGTTGGTGGTGAGGCCCTGGGCAGG30- example_title: CF transmembrane conductance regulator31  pipeline_tag: regulatory-track32  sequence_type: DNA33  task: regulatory-track34  text: ACTTCACTTCTAATGGTGATTATGGGAGAACTGGAGCCTTCAGAGGGTAAAATTAAGCACAGTGGAAGAATTTCATTCTGTTCTCAGTTTTCCTGGATTATGCCTGGCACCATTAAAGAAAATATCATCTTTGGTGTTTCCTATGATGAATATAGATACAGAAGCGTCATCAAAGCATGCCAACTAGAAGAG35- example_title: telomerase reverse transcriptase36  pipeline_tag: regulatory-track37  sequence_type: DNA38  task: regulatory-track39  text: CGCGGGGGTGGCCGGGGCCAGGGCTTCCCACGTGCGCAGCAGGACGCAGCGCTGCCTGAAACTCGCGCCGCGAGGAGAGGGCGGGGCCGCGGAAAGGAAGGGGAGGGGCTGGGAGGGCCCGGAGGGGGCTGGGCCGGGGACCCGGGAGGGGTCGGGACGGGGCGGGGTCCGCGCGGAGGAGGCGGAGCTGGAAGGTGAAGGGGCAGGACGGGTGCCCGGGTCCCCAGTCCCTCCGCCACGTGGGAAGCGCGGTCCTGGGCGTCTGTGCCCGCGAATCCACTGGGAGCCCGGCCTGGCCCCGACAGCGCAGCTGCTCCGGGCGGACCCGGGG40- example_title: KRAS proto-oncogene41  pipeline_tag: regulatory-track42  sequence_type: DNA43  task: regulatory-track44  text: GCCTGCTGAAAATGACTGAATATAAACTTGTGGTAGTTGGAGCTGGTGGCGTAGGCAAGAGTGCCTTGACGATACAGCTAATTCAGAATCATTTTGTGGACGAATATGATCCAACAATAGAG45- example_title: prion protein (Kanno blood group)46  pipeline_tag: regulatory-track47  sequence_type: cDNA48  task: regulatory-track49  text: ATGGCGAACCTTGGCTGCTGGATGCTGGTTCTCTTTGTGGCCACATGGAGTGACCTGGGCCTCTGC50- example_title: interleukin 1051  pipeline_tag: regulatory-track52  sequence_type: cDNA53  task: regulatory-track54  text: ATGCACAGCTCAGCACTGCTCTGTTGCCTGGTCCTCCTGACTGGGGTGAGGGCC55- example_title: Zaire ebolavirus56  pipeline_tag: regulatory-track57  sequence_type: cDNA58  task: regulatory-track59  text: AATGTTCAAACACTTTGTGAAGCTCTGTTAGCTGATGGTCTTGCTAAAGCATTTCCTAGCAATATGATGGTAGTCACAGAGCGTGAGCAAAAAGAAAGCTTATTGCATCAAGCATCATGGCACCACACAAGTGATGATTTTGGTGAGCATGCCACAGTTAGAGGGAGTAGCTTTGTAACTGATTTAGAGAAATACAATCTTGCATTTAGATATGAGTTTACAGCACCTTTTATAGAATATTGTAACCGTTGCTATGGTGTTAAGAATGTTTTTAATTGGATGCATTATACAATCCCACAGTGTTAT60- example_title: SARS coronavirus61  pipeline_tag: regulatory-track62  sequence_type: cDNA63  task: regulatory-track64  text: ATGTTTATTTTCTTATTATTTCTTACTCTCACTAGTGGTAGTGACCTTGACCGGTGCACCACTTTTGATGATGTTCAAGCTCCTAATTACACTCAACATACTTCATCTATGAGGGGGGTTTACTATCCTGATGAAATTTTTAGATCAGACACTCTTTATTTAACTCAGGATTTATTTCTTCCATTTTATTCTAATGTTACAGGGTTTCATACTATTAATCATACGTTTGACAACCCTGTCATACCTTTTAAGGATGGTATTTATTTTGCTGCCACAGAGAAATCAAATGTTGTCCGTGGTTGGGTTTTTGGTTCTACCATGAACAACAAGTCACAGTCGGTGATTATTATTAACAATTCTACTAATGTTGTTATACGAGCATGTAACTTTGAATTGTGTGACAACCCTTTCTTTGCTGTTTCTAAACCCATGGGTACACAGACACATACTATGATATTCGATAATGCATTTAAATGCACTTTCGAGTACATATCT65- example_title: insulin66  pipeline_tag: regulatory-track67  sequence_type: cDNA68  task: regulatory-track69  text: ATGGCCCTGTGGATGCGCCTCCTGCCCCTGCTGGCGCTGCTGGCCCTCTGGGGACCTGACCCAGCCGCAGCCTTTGTGAACCAACACCTGTGCGGCTCACACCTGGTGGAAGCTCTCTACCTAGTGTGCGGGGAACGAGGCTTCTTCTACACACCCAAGACCCGCCGGGAGGCAGAGGACCTGCAGGTGGGGCAGGTGGAGCTGGGCGGGGGCCCTGGTGCAGGCAGCCTGCAGCCCTTGGCCCTGGAGGGGTCCCTGCAGAAGCGTGGCATTGTGGAACAATGCTGTACCAGCATCTGCTCCCTCTACCAGCTGGAGAACTACTGCAACTAG70- example_title: cyclin dependent kinase inhibitor 2A71  pipeline_tag: regulatory-track72  sequence_type: cDNA73  task: regulatory-track74  text: ATGGAGCCGGCGGCGGGGAGCAGCATGGAGCCTTCGGCTGACTGGCTGGCCACGGCCGCGGCCCGGGGTCGGGTAGAGGAGGTGCGGGCGCTGCTGGAGGCGGGGGCGCTGCCCAACGCACCGAATAGTTACGGTCGGAGGCCGATCCAGGTCATGATGATGGGCAGCGCCCGAGTGGCGGAGCTGCTGCTGCTCCACGGCGCGGAGCCCAACTGCGCCGACCCCGCCACTCTCACCCGACCCGTGCACGACGCTGCCCGGGAGGGCTTCCTGGACACGCTGGTGGTGCTGCACCGGGCCGGGGCGCGGCTGGACGTGCGCGATGCCTGGGGCCGTCTGCCCGTGGACCTGGCTGAGGAGCTGGGCCATCGCGATGTCGCACGGTACCTGCGCGCGGCTGCGGGGGGCACCAGAGGCAGTAACCATGCCCGCATAGATGCCGCGGAAGGTCCCTCAGACATCCCCGATTGA75- example_title: human papillomavirus type 16 E676  pipeline_tag: regulatory-track77  sequence_type: cDNA78  task: regulatory-track79  text: ATGCACCAAAAGAGAACTGCAATGTTTCAGGACCCACAGGAGCGACCCAGAAAGTTACCACAGTTATGCACAGAGCTGCAAACAACTATACATGATATAATATTAGAATGTGTGTACTGCAAGCAACAGTTACTGCGACGTGAGGTATATGACTTTGCTTTTCGGGATTTATGCATAGTATATAGAGATGGGAATCCATATGCTGTATGTGATAAATGTTTAAAGTTTTATTCTAAAATTAGTGAGTATAGACATTATTGTTATAGTTTGTATGGAACAACATTAGAACAGCAATACAACAAACCGTTGTGTGATTTGTTAATTAGGTGTATTAACTGTCAAAAGCCACTGTGTCCTGAAGAAAAGCAAAGACATCTGGACAAAAAGCAAAGATTCCATAATATAAGGGGTCGGTGGACCGGTCGATGTATGTCTTGTTGCAGATCATCAAGAACACGTAGAGAAACCCAGCTGTAA80---81 82# Enformer83 84Transformer-based deep neural network for predicting genomic coverage tracks from long DNA sequences with long-range context.85 86## Disclaimer87 88This is an UNOFFICIAL implementation of [Effective gene expression prediction from sequence by integrating long-range interactions](https://doi.org/10.1038/s41592-021-01252-x) by Žiga Avsec, Vikram Agarwal, Daniel Visentin, et al.89 90The OFFICIAL repository of Enformer is at [google-deepmind/deepmind-research/enformer](https://github.com/google-deepmind/deepmind-research/tree/master/enformer).91 92> [!TIP]93> The MultiMolecule team has confirmed that the provided model and checkpoints are producing the same intermediate representations as the original implementation.94 95**The team releasing Enformer did not write this model card for this model so this model card has been written by the MultiMolecule team.**96 97## Model Details98 99Enformer is the successor of Basenji. It replaces Basenji's dilated convolution tower with a convolution stem followed by a Transformer trunk, which lets it model long-range genomic interactions. It consumes a long DNA window (~197 kb), passes it through a convolution + attention-pooling stem that downsamples the sequence by `2 ** 7 = 128x`, processes the binned representation with 11 Transformer blocks using Transformer-XL style relative positional encoding, center-crops to 896 output bins, and applies a pointwise head plus a per-species linear track projection with a softplus activation. The prediction is **binned**: the output has shape `(batch_size, target_length, num_tracks)` where each bin summarizes 128 bp of sequence and `num_tracks` is the number of genomic coverage experiments for the selected species.100 101### Model Specification102 103| Input Length | Bin Size | Output Bins | Hidden Size | Layers | Heads | Num Labels | Num Parameters (M) | FLOPs (P) | MACs (P) | Max Num Tokens |104| ------------ | -------- | ----------- | ----------- | ------ | ----- | ---------- | ------------------ | --------- | -------- | -------------- |105| 196608       | 128      | 896         | 1536        | 11     | 8     | 5313       | 246.18             | -         | -        | 196,608        |106 107The table reports the human output head. The mouse head predicts 1643 tracks.108FLOPs and MACs have not been recomputed for the canonical 196,608 bp Enformer input window.109 110### Links111 112- **Code**: [multimolecule.enformer](https://github.com/DLS5-Omics/multimolecule/tree/master/multimolecule/models/enformer)113- **Data**: ENCODE, FANTOM5, GTEx CAGE, ChIP-seq, DNase-seq, and related genomic coverage tracks114- **Paper**: [Effective gene expression prediction from sequence by integrating long-range interactions](https://doi.org/10.1038/s41592-021-01252-x)115- **Developed by**: Žiga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R. Ledsam, Agnieszka Grabska-Barwinska, Kyle R. Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, David R. Kelley116- **Model type**: Convolutional stem followed by Transformer trunk with long-range attention for binned multi-track genomic coverage prediction117- **Original Repository**: [google-deepmind/deepmind-research/enformer](https://github.com/google-deepmind/deepmind-research/tree/master/enformer)118 119## Usage120 121The model file depends on the [`multimolecule`](https://multimolecule.danling.org) library. You can install it using pip:122 123```bash124pip install multimolecule125```126 127### Direct Use128 129#### Genomic Coverage Prediction130 131You can use this model to predict binned genomic coverage tracks from a DNA sequence:132 133```python134>>> import torch135>>> from multimolecule import DnaTokenizer, EnformerConfig, EnformerForTokenPrediction136 137>>> config = EnformerConfig(138...     sequence_length=256, hidden_size=12, num_hidden_layers=1, num_attention_heads=2,139...     attention_head_size=4, num_downsamples=3, dim_divisible_by=2, target_length=16,140...     num_labels=4,141... )142>>> model = EnformerForTokenPrediction(config)143>>> output = model(torch.randint(config.vocab_size, (1, 256)))144>>> output.logits.shape145torch.Size([1, 16, 4])146>>> coverage, channels = model.postprocess(output)147>>> coverage.shape148torch.Size([1, 16, 4])149```150 151The binned positional axis is treated as the "token" axis: each output position corresponds to one152genomic bin rather than a single nucleotide. The `species` configuration option selects the153`human` (5,313 tracks) or `mouse` (1,643 tracks) output head.154 155### Interface156 157- **Input length**: fixed 196,608 bp DNA window158- **Output binning**: 128 bp per output bin; 896 output bins per window (after center-cropping the binned representation)159- **Species head**: select `human` (5,313 tracks) or `mouse` (1,643 tracks) via the `species` config option160- **Output**: raw pre-softplus `logits` of shape `(batch_size, target_length, num_tracks)`; use `postprocess` for non-negative coverage tracks161 162## Training Details163 164Enformer was trained to predict genomic coverage tracks (DNase-seq, ATAC-seq, ChIP-seq and CAGE)165from the human and mouse reference genomes.166 167### Training Data168 169The model was trained on a large compendium of functional genomics experiments aligned to the170human (hg38) and mouse (mm10) reference genomes. The genome was divided into overlapping windows;171for each window the per-128-bp coverage of every experiment served as the regression target.172 173### Training Procedure174 175#### Pre-training176 177The model was trained to minimize a Poisson regression loss between predicted and observed178coverage, using a softplus output activation to keep the predicted coverage non-negative.179 180## Citation181 182```bibtex183@article{avsec2021effective,184  author    = {Avsec, {\v{Z}}iga and Agarwal, Vikram and Visentin, Daniel and Ledsam, Joseph R. and Grabska-Barwinska, Agnieszka and Taylor, Kyle R. and Assael, Yannis and Jumper, John and Kohli, Pushmeet and Kelley, David R.},185  title     = {Effective gene expression prediction from sequence by integrating long-range interactions},186  journal   = {Nature Methods},187  year      = 2021,188  volume    = 18,189  number    = 10,190  pages     = {1196--1203},191  doi       = {10.1038/s41592-021-01252-x},192  publisher = {Nature Publishing Group}193}194```195 196> [!NOTE]197> The artifacts distributed in this repository are part of the MultiMolecule project.198> If MultiMolecule supports your research, please cite the MultiMolecule project as follows:199 200```bibtex201@software{chen_2024_12638419,202  author    = {Chen, Zhiyuan and Zhu, Sophia Y.},203  title     = {MultiMolecule},204  doi       = {10.5281/zenodo.12638419},205  publisher = {Zenodo},206  url       = {https://doi.org/10.5281/zenodo.12638419},207  year      = 2024,208  month     = may,209  day       = 4210}211```212 213## Contact214 215Please use GitHub issues of [MultiMolecule](https://github.com/DLS5-Omics/multimolecule/issues) for any questions or comments on the model card.216 217Please contact the authors of the [Enformer paper](https://doi.org/10.1038/s41592-021-01252-x) for questions or comments on the paper/model.218 219## License220 221This model implementation is licensed under the [GNU Affero General Public License](license.md).222 223For additional terms and clarifications, please refer to our [License FAQ](license-faq.md).224 225```spdx226SPDX-License-Identifier: AGPL-3.0-or-later227```