duttaprat/HViLM-base

HViLM-base: A Foundation Model for Viral Genomics

<div align="center">

   

</div>

[!IMPORTANT] HVUE v2 supersedes the original HVUE benchmark. The original HVUE v1 benchmark contained substantial cross-split sequence similarity that could inflate held-out performance estimates. HVUE v2 was rebuilt using source-sequence clustering before train/validation/test assignment and chunking, followed by exact- and near-match leakage auditing. Use [duttaprat/HVUE-v2](https://huggingface.co/datasets/duttaprat/HVUE-v2) for current HViLM evaluation.

Model Description

HViLM (Human Virome Language Model) is a genomic foundation model adapted to virus sequences through continued pre-training of DNABERT-2. HViLM-base was trained on approximately 5 million non-redundant virus-derived sequence fragments from the VIRION resource, representing approximately 9,000 virus species across 45+ families.

The architecture and tokenizer remain those of DNABERT-2; continued pre-training updates the model weights using a masked-language-modeling objective on the virus-focused corpus.

Preprint: HViLM: A Foundation Model for Viral Genomics Enables Multi-Task Prediction of Pathogenicity, Transmissibility, and Host Tropism bioRxiv 2026.03.18.712700

Authors: Pratik Dutta, Jack Vaska, Pallavi Surana, Rekha Sathian, Max Chao, Zhihan Zhou, Han Liu, and Ramana V. Davuluri

Code: github.com/duttaprat/HViLM

HViLM Model Family

HViLM-base is the continued-pretrained foundation model. Official task-specific models fine-tuned on HVUE v2 are released as standalone checkpoints:

The complete project is also grouped in the HViLM: Human Virome Language Model collection on the duttaprat Collections page.

Key Features

• —Virus-focused continued pre-training: approximately 5M non-redundant fragments derived from VIRION-linked virus sequences.
• —DNABERT-2 initialization: preserves the DNABERT-2 architecture and BPE tokenizer while adapting model weights to virus sequence data.
• —Three official downstream models: pathogenicity, transmissibility, and host tropism.
• —HVUE v2 evaluation: cluster-aware splitting before chunking, with multiple similarity stringencies and sequence lengths.
• —Parameter-efficient downstream adaptation: official task models were trained with LoRA.
• —Public reproducibility resources: base model, three task-specific checkpoints, HVUE v2 benchmark, and project code are released publicly.

Model Architecture and Continued Pre-training

HViLM-base is derived from DNABERT-2 (117M parameters).

Sequence-length note: HViLM uses BPE tokenization, so nucleotide length and model-token length are not equivalent. The continued-pretraining corpus used 1000-nt sequence fragments; downstream configurations are described by nucleotide length in HVUE v2.

Quick Start

Extract sequence representations from HViLM-base

import torch
from transformers import AutoTokenizer, AutoModel

model_id = "duttaprat/HViLM-base"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModel.from_pretrained(
    model_id,
    trust_remote_code=True,
)

sequence = "ATGCGTACGTTAGCCGATCGATTACGCGTACGTAGCTAGCTAGCT"

inputs = tokenizer(
    sequence,
    return_tensors="pt",
    truncation=True,
    padding=True,
)

with torch.no_grad():
    outputs = model(**inputs)

token_embeddings = outputs.last_hidden_state
print(token_embeddings.shape)

For sequence-level representations, pooling strategy should be chosen according to the downstream task rather than treated as a fixed property of HViLM-base.

Use the Official Fine-tuned Models

If the goal is one of the three HVUE v2 tasks, users can load the corresponding task model directly; HViLM-base does not need to be loaded separately.

Pathogenicity

from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "duttaprat/HViLM-Patho"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForSequenceClassification.from_pretrained(
    model_id,
    trust_remote_code=True,
)

Labels:

• —0 → NON_PATHOGENIC
• —1 → PATHOGENIC

Transmissibility

Use:

duttaprat/HViLM-R0

Labels:

• —0 → R0_LT_1
• —1 → R0_GE_1

Host Tropism

Use:

duttaprat/HViLM-Tropism

Labels:

• —0 → NON_HUMAN_TROPIC
• —1 → HUMAN_TROPIC

See the individual model cards for full usage examples and task-specific limitations.

HVUE v2 Benchmark

HVUE v2 is the current evaluation benchmark for HViLM. It replaces HVUE v1.

The benchmark was reconstructed to reduce supervised train-test leakage using the following ordering:

1. 1.consolidate and deduplicate source sequences;
2. 2.cluster source sequences with MMseqs2;
3. 3.assign complete clusters to train/validation/test splits;
4. 4.chunk sequences only after split assignment;
5. 5.remove exact duplicate chunks;
6. 6.audit cross-split exact and near matches.

HVUE v2 includes:

• —Pathogenicity
• —Transmissibility
• —Host Tropism

Across the benchmark, configurations evaluate different sequence lengths (500, 1000, and 2000 nt where applicable), sequence-similarity stringencies, and temporal generalization where reliable collection-date metadata are available.

Primary HViLM Results

The primary results below use the standard 1000-nt configuration for each task.

The directory/configuration identifiers retain bp for release stability; manuscript and descriptive text use nt for nucleotide sequence length.

Interpretation of the HVUE v2 Results

Under leakage-controlled evaluation, the effect of virus-focused continued pre-training is task dependent:

• —Pathogenicity: HViLM improves F1 by 1.28 points over vanilla DNABERT-2 (91.32 vs. 90.04).
• —Transmissibility: HViLM and DNABERT-2 are close (86.16 vs. 85.81 F1), and HViLM is essentially tied with DNABERT-MB (86.16 vs. 86.15 F1).
• —Host Tropism: HViLM shows the largest F1 improvement, reaching 74.49 compared with 64.82 for class-balanced DNABERT-2.

These results support a more specific conclusion than the original HVUE v1 evaluation: virus-focused continued pre-training provides its clearest benefit on the more challenging Host Tropism task, while gains on Pathogenicity and Transmissibility are smaller.

For complete baseline comparisons, hard-split evaluations, temporal evaluations, and sequence-length analyses, see the HViLM GitHub repository and HVUE-v2.

Training Data

Continued-pretraining corpus

HViLM-base was trained using virus sequences associated with the VIRION resource.

Processing included:

• —retrieval and quality control of VIRION-linked nucleotide sequences;
• —removal of short sequences and exact duplicates;
• —segmentation into non-overlapping 1000-nt fragments;
• —MMseqs2 clustering at 80% sequence identity and 80% coverage;
• —selection of approximately 5M representative fragments for continued pre-training.

The corpus spans approximately 9,000 virus species and 45+ virus families across the Baltimore classification groups.

Interpretability

Attention-guided analyses associated with the HViLM study identified candidate sequence motifs in pathogenic coronavirus sequences, including motifs with similarity to vertebrate transcription-factor binding motifs such as IRF1, FOXQ1, and ZNF354A.

These observations are hypothesis-generating. Sequence similarity between virus motifs and host transcription-factor binding motifs does not by itself establish molecular mimicry, causal regulation, immune evasion, or another biological mechanism. Experimental validation and additional controls are required for mechanistic interpretation.

Limitations

• —HVUE v2 controls supervised split leakage through source-level clustering and auditing, but sequence-similarity thresholds cannot eliminate every form of biological relatedness.
• —The complete historical training exposure of the original DNABERT-2 model cannot be reconstructed; therefore, absence of all possible ancestral pretraining exposure to benchmark-related sequences cannot be guaranteed.
• —Host association is biologically context-dependent and may include multi-host, zoonotic, and reverse-zoonotic relationships; the benchmark uses a simplified binary formulation.
• —R₀-based transmissibility labels simplify a continuous, context-dependent epidemiological quantity into a binary benchmark task.
• —Performance differences between closely matched models should not be interpreted as statistically meaningful without uncertainty estimates or repeated evaluations.
• —Attention-based motif analyses should be considered exploratory rather than direct evidence of mechanism.
• —HViLM predictions are research outputs and are not intended to replace experimental, clinical, epidemiological, or public-health assessment.

Reproducibility and Resources

• —Base model: duttaprat/HViLM-base
• —Pathogenicity model: duttaprat/HViLM-Patho
• —Transmissibility model: duttaprat/HViLM-R0
• —Host Tropism model: duttaprat/HViLM-Tropism
• —Benchmark: duttaprat/HVUE-v2
• —Code: github.com/duttaprat/HViLM
• —Collections: duttaprat's Hugging Face Collections

Citation

If you use HViLM in your research, please cite:

@article{dutta2026hvilm,
  title={HViLM: A foundation model for viral genomics enables multi-task prediction of pathogenicity, transmissibility, and host tropism},
  author={Dutta, Pratik and Vaska, Jack and Surana, Pallavi and Sathian, Rekha and Chao, Max and Zhou, Zhihan and Liu, Han and Davuluri, Ramana V},
  journal={bioRxiv},
  pages={2026--03},
  year={2026},
  publisher={Cold Spring Harbor Laboratory}
}

If you use DNABERT-2 directly or build on its architecture, please also cite the DNABERT-2 publication.

Model Card Authors

• —Pratik Dutta — Stony Brook University
• —Ramana V. Davuluri — Stony Brook University

Contact

• —GitHub Issues: github.com/duttaprat/HViLM/issues
• —Lab: Davuluri Lab, Stony Brook University

Acknowledgments

HViLM builds on DNABERT-2 by Zhou et al. Continued-pretraining data were derived from the VIRION resource maintained by the Viral Emergence Research Initiative (Verena).

License

HViLM-base is released under the Apache License 2.0.

Disclaimer

HViLM is a research model for computational biology. It should not be used as the sole basis for clinical, diagnostic, epidemiological, biosurveillance, or public-health decisions. Model outputs should be interpreted alongside appropriate biological evidence and expert assessment.