cisco-ai/SecureBERT2.0-biencoder

Model Card for cisco-ai/SecureBERT2.0-biencoder

The SecureBERT 2.0 Bi-Encoder is a cybersecurity-domain sentence-similarity and document-embedding model fine-tuned from SecureBERT 2.0. It independently encodes queries and documents into a shared vector space for semantic search, information retrieval, and cybersecurity knowledge retrieval.

Model Details

Model Description

• —Developed by: Cisco AI
• —Model type: Bi-Encoder (Sentence Transformer)
• —Architecture: ModernBERT backbone with dual encoders
• —Max sequence length: 1024 tokens
• —Output dimension: 768
• —Language: English
• —License: Apache-2.0
• —Finetuned from: cisco-ai/SecureBERT2.0-base

Uses

Direct Use

• —Semantic search and document similarity in cybersecurity corpora
• —Information retrieval and ranking for threat intelligence reports, advisories, and vulnerability notes
• —Document embedding for retrieval-augmented generation (RAG) and clustering

Downstream Use

• —Threat intelligence knowledge graph construction
• —Cybersecurity QA and reasoning systems
• —Security operations center (SOC) data mining

Out-of-Scope Use

• —Non-technical or general-domain text similarity
• —Generative or conversational tasks

Model Architecture

The Bi-Encoder encodes queries and documents independently into a joint vector space. This architecture enables scalable approximate nearest-neighbor search for candidate retrieval and semantic ranking.

Datasets

Fine-Tuning Datasets

Dataset Descriptions

• —Cybersecurity QA corpus: 43 k question–answer pairs, reports, and technical documents covering network security, malware analysis, cryptography, and cloud security.
• —Security governance QA corpus: 60 k expert-curated governance and compliance QA pairs emphasizing clear, validated responses.
• —Cybersecurity instruction–response corpus: 25 k instructional pairs enabling reasoning and instruction-following.
• —Cybersecurity rules corpus: 5 k structured policy and guideline records used for evaluation.

How to Get Started with the Model

Using Sentence Transformers

pip install -U sentence-transformers

Run Model to Encode

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("cisco-ai/SecureBERT2.0-biencoder")

sentences = [
    "How would you use Amcache analysis to detect fileless malware?",
    "Amcache analysis provides forensic artifacts for detecting fileless malware ...",
    "To capture and display network traffic"
]

embeddings = model.encode(sentences)
print(embeddings.shape)

Compute Similarity

from sentence_transformers import util
similarity = util.cos_sim(embeddings, embeddings)
print(similarity)

Framework Versions

• —python: 3.10.10
• —sentence_transformers: 5.0.0
• —transformers: 4.52.4
• —PyTorch: 2.7.0+cu128
• —accelerate: 1.9.0
• —datasets: 3.6.0

Training Details

Example Schema

Example Samples

Training Objective and Loss

The model was optimized to maximize semantic similarity between relevant cybersecurity text pairs using contrastive learning.

• —Loss Function: MultipleNegativesRankingLoss

Loss Parameters

{
    "scale": 20.0,
    "similarity_fct": "cos_sim"
}

Reference

@article{aghaei2025securebert,
  title={SecureBERT 2.0: Advanced Language Model for Cybersecurity Intelligence},
  author={Aghaei, Ehsan and Jain, Sarthak and Arun, Prashanth and Sambamoorthy, Arjun},
  journal={arXiv preprint arXiv:2510.00240},
  year={2025}
}

Model Card Authors

Cisco AI

Model Card Contact

For inquiries, please contact ai-threat-intel@cisco.com