selfconstruct3d/AttackGroup-MPNET
317
1---2library_name: transformers3tags:4- cybersecurity5- mpnet6- classification7- fine-tuned8language:9- en10base_model:11- sentence-transformers/all-mpnet-base-v212---13 14# AttackGroup-MPNET - Model Card for MPNet Cybersecurity Classifier15 16This is a fine-tuned MPNet model specialized for classifying cybersecurity threat groups based on textual descriptions of their tactics and techniques.17 18## Model Details19 20### Model Description21 22This model is a fine-tuned MPNet classifier specialized in categorizing cybersecurity threat groups based on textual descriptions of their tactics, techniques, and procedures (TTPs).23 24- **Developed by:** Dženan Hamzić25- **Model type:** Transformer-based classification model (MPNet)26- **Language(s) (NLP):** English27- **License:** Apache-2.028- **Finetuned from model:** microsoft/mpnet-base (with intermediate MLM fine-tuning)29 30### Model Sources31 32- **Base Model:** [microsoft/mpnet-base](https://huggingface.co/microsoft/mpnet-base)33 34## Uses35 36### Direct Use37 38This model classifies textual cybersecurity descriptions into known cybersecurity threat groups.39 40### Downstream Use41 42Integration into Cyber Threat Intelligence platforms, SOC incident analysis tools, and automated threat detection systems.43 44### Out-of-Scope Use45 46- General language tasks unrelated to cybersecurity47- Tasks outside the cybersecurity domain48 49## Bias, Risks, and Limitations50 51This model specializes in cybersecurity contexts. Predictions for unrelated contexts may be inaccurate.52 53### Recommendations54 55Always verify predictions with cybersecurity analysts before using in critical decision-making scenarios.56 57## How to Get Started with the Model (Classification)58 59```python60import torch61import torch.nn as nn62from transformers import AutoTokenizer, AutoModelForSequenceClassification63import torch.optim as optim64import numpy as np65from huggingface_hub import hf_hub_download66import json67 68device = torch.device("cuda" if torch.cuda.is_available() else "cpu")69 70 71label_to_groupid_file = hf_hub_download(72 repo_id="selfconstruct3d/AttackGroup-MPNET",73 filename="label_to_groupid.json"74)75 76with open(label_to_groupid_file, "r") as f:77 label_to_groupid = json.load(f)78 79# Load explicitly your fine-tuned MPNet model80classifier_model = AutoModelForSequenceClassification.from_pretrained("selfconstruct3d/AttackGroup-MPNET", num_labels=len(label_to_groupid)).to(device)81 82# Load explicitly your tokenizer83tokenizer = AutoTokenizer.from_pretrained("selfconstruct3d/AttackGroup-MPNET")84 85def predict_group(sentence):86 classifier_model.eval()87 encoding = tokenizer(88 sentence,89 truncation=True,90 padding="max_length",91 max_length=128,92 return_tensors="pt"93 )94 input_ids = encoding["input_ids"].to(device)95 attention_mask = encoding["attention_mask"].to(device)96 97 with torch.no_grad():98 outputs = classifier_model(input_ids=input_ids, attention_mask=attention_mask)99 logits = outputs.logits100 predicted_label = torch.argmax(logits, dim=1).cpu().item()101 102 predicted_groupid = label_to_groupid[str(predicted_label)]103 return predicted_groupid104 105# Example usage explicitly:106sentence = "APT38 has used phishing emails with malicious links to distribute malware."107predicted_class = predict_group(sentence)108print(f"Predicted GroupID: {predicted_class}")109```110Predicted GroupID: G0001111https://attack.mitre.org/groups/G0001/112 113 114## How to Get Started with the Model (Embeddings)115 116```python117import torch118from transformers import AutoTokenizer, AutoModelForSequenceClassification119from huggingface_hub import hf_hub_download120import json121 122device = torch.device("cuda" if torch.cuda.is_available() else "cpu")123 124 125label_to_groupid_file = hf_hub_download(126 repo_id="selfconstruct3d/AttackGroup-MPNET",127 filename="label_to_groupid.json"128)129 130with open(label_to_groupid_file, "r") as f:131 label_to_groupid = json.load(f)132 133 134# Load your fine-tuned classification model135model_name = "selfconstruct3d/AttackGroup-MPNET"136tokenizer = AutoTokenizer.from_pretrained(model_name)137classifier_model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=len(label_to_groupid)).to(device)138 139def get_embedding(sentence):140 classifier_model.eval()141 142 encoding = tokenizer(143 sentence,144 truncation=True,145 padding="max_length",146 max_length=128,147 return_tensors="pt"148 )149 input_ids = encoding["input_ids"].to(device)150 attention_mask = encoding["attention_mask"].to(device)151 152 with torch.no_grad():153 outputs = classifier_model.mpnet(input_ids=input_ids, attention_mask=attention_mask)154 cls_embedding = outputs.last_hidden_state[:, 0, :].cpu().numpy().flatten()155 156 return cls_embedding157 158# Example explicitly:159sentence = "APT38 has used phishing emails with malicious links to distribute malware."160embedding = get_embedding(sentence)161print("Embedding shape:", embedding.shape)162print("Embedding values:", embedding)163```164 165 166 167## Training Details168 169### Training Data170 171To be anounced...172 173### Training Procedure174 175- Fine-tuned from: MLM fine-tuned MPNet ("mpnet_mlm_cyber_finetuned-v2")176- Epochs: 32177- Learning rate: 5e-6178- Batch size: 16179 180## Evaluation181 182### Testing Data, Factors & Metrics183 184- **Testing Data:** Stratified sample from original dataset.185- **Metrics:** Accuracy, Weighted F1 Score186 187### Results188 189| Metric | Value |190|------------------------|---------|191| Cl. Accuracy (Test) | 0.9564 |192| W. F1 Score (Test) | 0.9577 |193 194 195## Evaluation Results196 197| Model | Accuracy | F1 Macro | F1 Weighted | Embedding Variability |198|-----------------------|----------|----------|-------------|-----------------------|199| **AttackGroup-MPNET** | **0.85** | **0.759**| **0.847** | 0.234 |200| GTE Large | 0.66 | 0.571 | 0.667 | 0.266 |201| E5 Large v2 | 0.64 | 0.541 | 0.650 | 0.355 |202| Original MPNet | 0.63 | 0.534 | 0.619 | 0.092 |203| BGE Large | 0.53 | 0.418 | 0.519 | 0.366 |204| SupSimCSE | 0.50 | 0.373 | 0.479 | 0.227 |205| MLM Fine-tuned MPNet | 0.44 | 0.272 | 0.411 | 0.125 |206| SecBERT | 0.41 | 0.315 | 0.410 | 0.591 |207| SecureBERT_Plus | 0.36 | 0.252 | 0.349 | 0.267 |208| CySecBERT | 0.34 | 0.235 | 0.323 | 0.229 |209| ATTACK-BERT | 0.33 | 0.240 | 0.316 | 0.096 |210| Secure_BERT | 0.00 | 0.000 | 0.000 | 0.007 |211| CyBERT | 0.00 | 0.000 | 0.000 | 0.015 |212 213 214| Model | Similarity Search Recall@5 | Few-shot Accuracy | In-dist Similarity | OOD Similarity | Robustness Similarity |215|----------------------|----------------------------|-------------------|--------------------|----------------|-----------------------|216| **AttackGroup-MPNET**| **0.934** | **0.857** | 0.235 | 0.017 | 0.948 |217| Original MPNet | 0.786 | 0.643 | 0.217 | -0.004 | 0.941 |218| E5 Large v2 | 0.778 | 0.679 | 0.727 | 0.013 | 0.977 |219| GTE Large | 0.746 | 0.786 | 0.845 | 0.002 | 0.984 |220| BGE Large | 0.632 | 0.750 | 0.533 | -0.006 | 0.970 |221| SupSimCSE | 0.616 | 0.571 | 0.683 | -0.015 | 0.978 |222| SecBERT | 0.468 | 0.429 | 0.586 | -0.001 | 0.970 |223| CyBERT | 0.452 | 0.250 | 1.000 | -0.001 | 1.000 |224| ATTACK-BERT | 0.362 | 0.571 | 0.157 | -0.005 | 0.950 |225| CySecBERT | 0.424 | 0.500 | 0.734 | -0.015 | 0.954 |226| Secure_BERT | 0.424 | 0.250 | 0.990 | 0.050 | 0.998 |227| SecureBERT_Plus | 0.406 | 0.464 | 0.981 | 0.040 | 0.998 |228 229 230 231### Single Prediction Example232 233```python234 235import torch236import torch.nn as nn237from transformers import AutoTokenizer, AutoModelForSequenceClassification238import torch.optim as optim239import numpy as np240from huggingface_hub import hf_hub_download241import json242 243device = torch.device("cuda" if torch.cuda.is_available() else "cpu")244# Load explicitly your fine-tuned MPNet model245classifier_model = AutoModelForSequenceClassification.from_pretrained("selfconstruct3d/AttackGroup-MPNET").to(device)246 247# Load explicitly your tokenizer248tokenizer = AutoTokenizer.from_pretrained("selfconstruct3d/AttackGroup-MPNET")249 250 251label_to_groupid_file = hf_hub_download(252 repo_id="selfconstruct3d/AttackGroup-MPNET",253 filename="label_to_groupid.json"254)255 256with open(label_to_groupid_file, "r") as f:257 label_to_groupid = json.load(f)258 259def predict_group(sentence):260 classifier_model.eval()261 encoding = tokenizer(262 sentence,263 truncation=True,264 padding="max_length",265 max_length=128,266 return_tensors="pt"267 )268 input_ids = encoding["input_ids"].to(device)269 attention_mask = encoding["attention_mask"].to(device)270 271 with torch.no_grad():272 outputs = classifier_model(input_ids=input_ids, attention_mask=attention_mask)273 logits = outputs.logits274 predicted_label = torch.argmax(logits, dim=1).cpu().item()275 276 predicted_groupid = label_to_groupid[str(predicted_label)]277 return predicted_groupid278 279# Example usage explicitly:280sentence = "APT38 has used phishing emails with malicious links to distribute malware."281predicted_class = predict_group(sentence)282print(f"Predicted GroupID: {predicted_class}")283```284 285## Environmental Impact286 287Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute).288 289- **Hardware Type:** [To be filled by user]290- **Hours used:** [To be filled by user]291- **Cloud Provider:** [To be filled by user]292- **Compute Region:** [To be filled by user]293- **Carbon Emitted:** [To be filled by user]294 295## Technical Specifications296 297### Model Architecture298 299- MPNet architecture with classification head (768 -> 512 -> num_labels)300- Last 10 transformer layers fine-tuned explicitly301 302## Environmental Impact303 304Carbon emissions should be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute).305 306## Model Card Authors307 308- Dženan Hamzić309 310## Model Card Contact311 312- https://www.linkedin.com/in/dzenan-hamzic/313 314 315## Licence316This model is licensed for non-commercial use only (CC BY-NC 4.0).317For commercial inquiries, please contact dzenan.hamzic@ait.ac.at.