CIRCL/vulnerability-attack-technique-classification-roberta-base
1147
1---2library_name: transformers3license: cc-by-4.04base_model: roberta-base5pipeline_tag: text-classification6language:7- en8datasets:9- CIRCL/vulnerability-attack-techniques10tags:11- security12- vulnerability13- cve14- mitre-attack15- cti16- multi-label-classification17- generated_from_trainer18model-index:19- name: vulnerability-attack-technique-classification-roberta-base20 results:21 - task:22 type: text-classification23 name: Multi-label MITRE ATT&CK technique classification24 dataset:25 name: CIRCL/vulnerability-attack-techniques26 type: CIRCL/vulnerability-attack-techniques27 split: test28 metrics:29 - type: recall30 name: Recall@531 value: 0.644032 - type: recall33 name: Recall@334 value: 0.518135 - type: f136 name: F1 micro37 value: 0.389938 - type: f139 name: F1 macro40 value: 0.191041---42 43# vulnerability-attack-technique-classification-roberta-base44 45Suggests [MITRE ATT&CK](https://attack.mitre.org/) (Enterprise) techniques46from a free-text vulnerability description. This is a multi-label classifier47([roberta-base](https://huggingface.co/roberta-base) with a sigmoid head, one48output per technique) fine-tuned on49[CIRCL/vulnerability-attack-techniques](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques),50a gold dataset of 1,207 CVEs whose labels come from expert51[MITRE CTID](https://ctid.mitre.org/) mappings. Given a CVE description it52scores 53 parent techniques (e.g. T1190 *Exploit Public-Facing Application*,53T1505 *Server Software Component*), for use as a ranked list of candidate54techniques for analyst review.55 56The model is trained with [VulnTrain](https://github.com/vulnerability-lookup/VulnTrain)57and runs in production on the public [Vulnerability-Lookup](https://vulnerability.circl.lu)58instance operated by CIRCL, served locally by59[ML-Gateway](https://github.com/vulnerability-lookup/ML-Gateway): every60vulnerability page has an ATT&CK tab with the model's suggestions (example:61[CVE-2021-44077](https://vulnerability.circl.lu/vuln/CVE-2021-44077#attack)).62 63The methodology, evaluation protocol, and the negative result on64LLM-assisted label expansion are described in the paper65[*Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and66the Limits of LLM-Assisted Label Expansion*](https://arxiv.org/abs/2607.25572)67(arXiv:2607.25572).68 69DOI: [10.57967/hf/9623](https://doi.org/10.57967/hf/9623)70 71## How to use72 73With VulnTrain, against a live CVE:74 75```bash76vulntrain-infer-attack-classification --cve CVE-2021-44077 \77 --model CIRCL/vulnerability-attack-technique-classification-roberta-base78```79 80With plain Transformers:81 82```python83import torch84from transformers import AutoModelForSequenceClassification, AutoTokenizer85 86model_id = "CIRCL/vulnerability-attack-technique-classification-roberta-base"87tokenizer = AutoTokenizer.from_pretrained(model_id)88model = AutoModelForSequenceClassification.from_pretrained(model_id)89model.eval()90 91description = (92 "Zoho ManageEngine ServiceDesk Plus before 11306, ServiceDesk Plus MSP "93 "before 10530, and SupportCenter Plus before 11014 are vulnerable to "94 "unauthenticated remote code execution."95)96inputs = tokenizer(description, truncation=True, max_length=512, return_tensors="pt")97with torch.no_grad():98 probs = torch.sigmoid(model(**inputs).logits)[0]99 100for i in probs.argsort(descending=True)[:5]:101 print(f"{model.config.id2label[int(i)]} {probs[i]:.4f}")102```103 104Technique IDs map to names via the105[ATT&CK Enterprise matrix](https://attack.mitre.org/techniques/enterprise/);106`sigmoid >= 0.5` is the prediction threshold used in evaluation, but the107model is most useful as a ranker (see the recall@k numbers below).108 109## Intended uses & limitations110 111**Intended**: triage assistance — given a vulnerability description, surface112a short ranked list of candidate ATT&CK techniques for a human analyst to113confirm or reject. This is how it is deployed in Vulnerability-Lookup, where114the UI explicitly flags the output as unverified AI-generated suggestions.115 116**Limitations**:117 118- The label space is the 53 parent techniques with at least 5 training119 examples; sub-techniques are collapsed to their parent, and techniques120 outside this vocabulary can never be suggested.121- The gold set skews toward exploited-in-the-wild CVEs (CTID's corpus and122 the KEV catalog), so coverage is best for the techniques common there.123- Trained on 972 CVEs — deliberately label-quality-bound rather than124 data-bound (see the paper's gold-size scaling curve: every metric still125 improves monotonically with more curated rows).126- English descriptions only; input is truncated at 512 tokens.127- Scores are not calibrated probabilities.128- Suggestions are **not** verified mappings; treat them as guidance, never129 as authoritative CTI.130 131## Training and evaluation data132 133Labels come from the two public expert sources of CVE→ATT&CK mappings, both134following the CTID [*Mapping ATT&CK to CVE for135Impact*](https://ctid.mitre.org/projects/mapping-attck-to-cve-for-impact)136methodology: the CTID `attack_to_cve` mappings and the CTID Mappings137Explorer KEV mappings (ATT&CK 16.1). Technique IDs revoked since publication138are remapped to their successors via the ATT&CK STIX `revoked-by`139relationships. Descriptions are joined from140[CIRCL/vulnerability-scores](https://huggingface.co/datasets/CIRCL/vulnerability-scores)141(Vulnerability-Lookup). Full details in the142[dataset card](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques)143and the [VulnTrain documentation](https://github.com/vulnerability-lookup/VulnTrain/blob/main/docs/attack-techniques-dataset.md).144 145Splits: 972 train / 106 validation / 118 test examples. **Checkpoint146selection uses the validation split; the test split is touched once, for the147numbers reported here.** The paper documents why this matters: selecting the148best checkpoint on the test split inflates and destabilizes small-test-split149metrics enough to produce spurious conclusions.150 151Notably, this model was **not** trained on LLM-generated labels: the paper's152controlled experiments show that folding in LLM-labeled CVEs at ≈0.39153agreement with the experts yields no reliable ranking improvement and154measurably degrades rare-technique coverage at scale (the comparison155checkpoint is published as156[...-llm-expanded](https://huggingface.co/CIRCL/vulnerability-attack-technique-classification-roberta-base-llm-expanded)).157 158## Evaluation159 160Held-out test split, this checkpoint (seed 42):161 162| Metric | Value |163|--------|-------|164| Recall@5 | 0.6440 |165| Recall@3 | 0.5181 |166| F1 micro | 0.3899 |167| F1 macro | 0.1910 |168| Precision micro | 0.2740 |169| Recall micro | 0.6756 |170 171Recall@5 = 0.64 means that on average 64% of an unseen CVE's expert-assigned172techniques appear in the model's top five suggestions — roughly double the173zero-shot embedding-similarity baseline reported in the paper. Across five174seeds under the identical protocol, the numbers of record are recall@51750.673 ± 0.019, recall@3 0.536 ± 0.032, micro-F1 0.410 ± 0.006, macro-F11760.177 ± 0.014. The complete trainer logs are published in the177[paper repository](https://github.com/vulnerability-lookup/cve-attack-mapping-paper/tree/master/trainer-logs).178 179## Training procedure180 181Binary cross-entropy over 53 sigmoid outputs, with per-label `pos_weight`182balancing (capped at 20) to keep rare techniques trainable. Trained with183`vulntrain-train-attack-classification` (VulnTrain).184 185### Training hyperparameters186 187The following hyperparameters were used during training:188- learning_rate: 1e-05189- train_batch_size: 32190- eval_batch_size: 32191- seed: 42192- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments193- lr_scheduler_type: linear194- num_epochs: 40195- max_length: 512196- loss: BCEWithLogitsLoss, balanced pos_weight (min 2.447, max 20.0)197- checkpoint selection: best macro-F1 on the validation split198 199### Training results200 201| Training Loss | Epoch | Step | Validation Loss | F1 Micro | F1 Macro | Precision Micro | Recall Micro | Recall At 3 | Recall At 5 |202|:-------------:|:-----:|:----:|:---------------:|:--------:|:--------:|:---------------:|:------------:|:-----------:|:-----------:|203| 0.9311 | 1.0 | 31 | 0.8413 | 0.1833 | 0.0401 | 0.1258 | 0.3376 | 0.1798 | 0.2611 |204| 0.8366 | 2.0 | 62 | 0.7876 | 0.1452 | 0.0234 | 0.1065 | 0.2278 | 0.1047 | 0.2086 |205| 0.8116 | 3.0 | 93 | 0.7717 | 0.2054 | 0.0518 | 0.1641 | 0.2743 | 0.1938 | 0.3234 |206| 0.7941 | 4.0 | 124 | 0.7576 | 0.3204 | 0.0804 | 0.2382 | 0.4895 | 0.3387 | 0.5009 |207| 0.7743 | 5.0 | 155 | 0.7435 | 0.3044 | 0.0841 | 0.2119 | 0.5401 | 0.3313 | 0.4696 |208| 0.7645 | 6.0 | 186 | 0.7290 | 0.3208 | 0.0906 | 0.2174 | 0.6118 | 0.3969 | 0.5391 |209| 0.7472 | 7.0 | 217 | 0.7163 | 0.3551 | 0.1130 | 0.2571 | 0.5738 | 0.4068 | 0.5741 |210| 0.7219 | 8.0 | 248 | 0.7056 | 0.3224 | 0.1079 | 0.2173 | 0.6245 | 0.4079 | 0.5521 |211| 0.7195 | 9.0 | 279 | 0.6933 | 0.3576 | 0.1495 | 0.2449 | 0.6624 | 0.4252 | 0.5663 |212| 0.6835 | 10.0 | 310 | 0.6845 | 0.3705 | 0.1665 | 0.2579 | 0.6582 | 0.4708 | 0.6090 |213| 0.6539 | 11.0 | 341 | 0.6768 | 0.4063 | 0.1810 | 0.2947 | 0.6540 | 0.5227 | 0.6318 |214| 0.6484 | 12.0 | 372 | 0.6725 | 0.3632 | 0.1734 | 0.2520 | 0.6498 | 0.4449 | 0.6200 |215| 0.6249 | 13.0 | 403 | 0.6664 | 0.3974 | 0.1782 | 0.2862 | 0.6498 | 0.5034 | 0.6396 |216| 0.6109 | 14.0 | 434 | 0.6585 | 0.3801 | 0.1721 | 0.2724 | 0.6287 | 0.4834 | 0.6491 |217| 0.6004 | 15.0 | 465 | 0.6539 | 0.3872 | 0.1678 | 0.2781 | 0.6371 | 0.4752 | 0.6347 |218| 0.5896 | 16.0 | 496 | 0.6502 | 0.4049 | 0.1777 | 0.2996 | 0.6245 | 0.4768 | 0.6397 |219| 0.5667 | 17.0 | 527 | 0.6478 | 0.3866 | 0.1682 | 0.2737 | 0.6582 | 0.4941 | 0.6472 |220| 0.5661 | 18.0 | 558 | 0.6425 | 0.4108 | 0.1910 | 0.3022 | 0.6414 | 0.5128 | 0.6667 |221| 0.5501 | 19.0 | 589 | 0.6394 | 0.3880 | 0.1861 | 0.2758 | 0.6540 | 0.4822 | 0.6561 |222| 0.5461 | 20.0 | 620 | 0.6377 | 0.4097 | 0.1804 | 0.3010 | 0.6414 | 0.5069 | 0.6687 |223| 0.5351 | 21.0 | 651 | 0.6338 | 0.4028 | 0.1715 | 0.3002 | 0.6118 | 0.4987 | 0.6624 |224| 0.5215 | 22.0 | 682 | 0.6351 | 0.4146 | 0.1964 | 0.3054 | 0.6456 | 0.4943 | 0.6875 |225| 0.5155 | 23.0 | 713 | 0.6315 | 0.4056 | 0.1737 | 0.3023 | 0.6160 | 0.4994 | 0.6553 |226| 0.5063 | 24.0 | 744 | 0.6269 | 0.4286 | 0.1879 | 0.3208 | 0.6456 | 0.5195 | 0.6923 |227| 0.5061 | 25.0 | 775 | 0.6264 | 0.4178 | 0.1869 | 0.3069 | 0.6540 | 0.5246 | 0.6656 |228| 0.4996 | 26.0 | 806 | 0.6301 | 0.4073 | 0.1792 | 0.3053 | 0.6118 | 0.5274 | 0.6958 |229| 0.4950 | 27.0 | 837 | 0.6225 | 0.4133 | 0.1771 | 0.3079 | 0.6287 | 0.5376 | 0.6593 |230| 0.4928 | 28.0 | 868 | 0.6228 | 0.4173 | 0.1768 | 0.3166 | 0.6118 | 0.5187 | 0.7048 |231| 0.4819 | 29.0 | 899 | 0.6242 | 0.4263 | 0.1871 | 0.3225 | 0.6287 | 0.5494 | 0.6970 |232| 0.4766 | 30.0 | 930 | 0.6194 | 0.4166 | 0.1862 | 0.3094 | 0.6371 | 0.5226 | 0.6871 |233| 0.4694 | 31.0 | 961 | 0.6213 | 0.42 | 0.1914 | 0.3175 | 0.6203 | 0.5399 | 0.6918 |234| 0.4781 | 32.0 | 992 | 0.6209 | 0.4292 | 0.1893 | 0.3281 | 0.6203 | 0.5439 | 0.7060 |235| 0.4636 | 33.0 | 1023 | 0.6218 | 0.4347 | 0.1962 | 0.3276 | 0.6456 | 0.525 | 0.6797 |236| 0.4641 | 34.0 | 1054 | 0.6216 | 0.4314 | 0.1887 | 0.3261 | 0.6371 | 0.5415 | 0.6797 |237| 0.4592 | 35.0 | 1085 | 0.6206 | 0.4313 | 0.1916 | 0.3282 | 0.6287 | 0.5466 | 0.6858 |238| 0.4526 | 36.0 | 1116 | 0.6208 | 0.4357 | 0.1926 | 0.3333 | 0.6287 | 0.5447 | 0.6863 |239| 0.4643 | 37.0 | 1147 | 0.6191 | 0.4218 | 0.1899 | 0.3196 | 0.6203 | 0.5392 | 0.6863 |240| 0.4501 | 38.0 | 1178 | 0.6191 | 0.4242 | 0.1885 | 0.3224 | 0.6203 | 0.5368 | 0.6929 |241| 0.4570 | 39.0 | 1209 | 0.6206 | 0.4350 | 0.1932 | 0.3326 | 0.6287 | 0.5281 | 0.6910 |242| 0.4436 | 40.0 | 1240 | 0.6199 | 0.4325 | 0.1916 | 0.3296 | 0.6287 | 0.5329 | 0.6882 |243 244### Framework versions245 246- Transformers 5.13.0247- Pytorch 2.12.1+cu130248- Datasets 4.8.5249- Tokenizers 0.22.2250 251## Related artifacts252 253| Artifact | Location | DOI |254|----------|----------|-----|255| Gold dataset (1,207 CVEs, CTID-curated labels) | [CIRCL/vulnerability-attack-techniques](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques) | [10.57967/hf/9621](https://doi.org/10.57967/hf/9621) |256| LLM expansion dataset (negative result) | [CIRCL/vulnerability-attack-techniques-llm-scaling](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques-llm-scaling) | [10.57967/hf/9622](https://doi.org/10.57967/hf/9622) |257| LLM-expanded comparison model | [CIRCL/vulnerability-attack-technique-classification-roberta-base-llm-expanded](https://huggingface.co/CIRCL/vulnerability-attack-technique-classification-roberta-base-llm-expanded) | [10.57967/hf/9624](https://doi.org/10.57967/hf/9624) |258| Code | [vulnerability-lookup/VulnTrain](https://github.com/vulnerability-lookup/VulnTrain) | — |259| Paper | [arXiv:2607.25572](https://arxiv.org/abs/2607.25572) | — |260| Paper LaTeX source + trainer logs | [vulnerability-lookup/cve-attack-mapping-paper](https://github.com/vulnerability-lookup/cve-attack-mapping-paper) | — |261 262## Citation263 264```bibtex265@misc{bonhomme2026mappingcvesmitreattck,266 title={Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion},267 author={Cédric Bonhomme and Alexandre Dulaunoy},268 year={2026},269 eprint={2607.25572},270 archivePrefix={arXiv},271 primaryClass={cs.CR},272 url={https://arxiv.org/abs/2607.25572},273}274```275 276## Acknowledgements277 278Developed at [CIRCL](https://www.circl.lu) in the context of the279[AIPITCH](https://www.science.nask.pl/en/research-areas/projects/12456)280project, co-funded by the European Union.281