ctokx/tram-attack-multilabel-clean

TRAM ATT&CK Multi-Label (cleaned, with leak-free splits)

Sentence-level multi-label mapping of cyber threat intelligence prose to MITRE ATT&CK technique IDs. Derived from MITRE CTID's TRAM corpus, deduplicated and republished with two split schemes so that leakage can be measured rather than assumed.

Everything here is regenerated by python scripts/01_build_dataset.py. No row was edited by hand.

Why this exists

The upstream corpus contains 19,178 sentences drawn from only 151 source reports. Threat reports repeat phrasing heavily, both within a report and across reports covering the same campaign, so the sentence-level random split that most work uses puts near-identical prose on both sides of the evaluation.

This release ships both splits and expects you to report both:

Construction

Deduplication takes the union of labels across duplicates. Two copies of one sentence annotated [T1027] and [T1027, T1140] are one sentence annotated inconsistently; keeping the union recovers the label instead of discarding it with the duplicate row. The surviving row keeps the first document it appeared in, so no sentence can span two documents.

T1557.001 is dropped because it appears in exactly one document. It cannot be placed in train and test, so it is either untrainable or unevaluable. Dropping it and saying so beats reporting F1 = 0.00 for it.

Splits

The document split is built by group-aware multi-label stratification in two phases. A plain greedy pass fails here: with a 70/15/15 target the train split always shows the largest absolute label deficit, so it absorbs every document containing a rare technique and the test set ends up with none. Phase 1 therefore reserves one document per technique for each split, rarest technique first; phase 2 packs the remainder using deficit normalised by split size.

Consequences, both worth knowing:

• —All 49 techniques appear in all three `document` splits. The naive random split does not manage this. T1072 has zero dev examples, so its threshold cannot be tuned.
• —Coverage seeding costs some size balance. The document split lands at roughly 63/17/20 rather than 70/15/15. Larger dev and test sets make the estimates more stable, so this was accepted rather than corrected.

Fields

{
  "sentence":  "The dropper base64-encodes its configuration before writing it to disk.",
  "labels":    ["T1027"],
  "doc_title": "NotPetya Technical Analysis  A Triple Threat ..."
}

labels is empty for 78.2% of rows. These are real negatives and should be kept. A detector that fires on every sentence of a threat report is useless to an analyst.

Known limitations

• —50 techniques, not the full ATT&CK matrix. The upstream corpus covers the 49 retained here. A technique outside that set will never be predicted.
• —Heavy long tail. T1027 has 678 instances; the rarest retained techniques have roughly 20. Macro-averaged metrics are the meaningful ones.
• —Single-sentence context. Labels were assigned per sentence, so techniques only inferable from surrounding paragraphs are under-represented.
• —151 documents is not many. Even the document split is one draw from a small pool; treat differences of a point or two as noise.
• —Annotation is not exhaustive. Some unlabelled sentences do describe techniques. Absolute recall is therefore pessimistic.

Licence and provenance

Derived from the TRAM corpus, released by the MITRE Center for Threat-Informed Defense under Apache-2.0; this derivative keeps Apache-2.0. The underlying sentences are fragments of publicly-published vendor threat reports as redistributed by MITRE CTID.

Technique names come from MITRE ATT&CK STIX data, used under the ATT&CK Terms of Use. ATT&CK® is a registered trademark of The MITRE Corporation.

AnnoCTR was deliberately not merged in. It is CC-BY-SA 4.0, and share-alike would relicense this entire dataset and restrict downstream reuse. It remains a good candidate for a separately-licensed evaluation split.

Citation

@misc{tram_attack_multilabel_clean,
  title  = {TRAM ATT&CK Multi-Label (cleaned, leak-free splits)},
  author = {Varol Cagdas Tok},
  year   = {2026},
  url    = {https://huggingface.co/datasets/ctokx/tram-attack-multilabel-clean},
  note   = {Derived from MITRE CTID TRAM, Apache-2.0}
}

Please also cite the upstream corpus:

@misc{tram,
  title        = {TRAM: Threat Report ATT&CK Mapper},
  author       = {{MITRE Center for Threat-Informed Defense}},
  howpublished = {\url{https://github.com/center-for-threat-informed-defense/tram}}
}