THULab/Advanced_SIEM_Dataset

Advanced SIEM Dataset (TsFile)

This dataset is a lossless conversion to the [Apache TsFile](https://tsfile.apache.org/) format of the HuggingFace dataset `darkknight25/Advanced_SIEM_Dataset`: a synthetic SIEM (Security Information and Event Management) event log for cybersecurity ML/AI research.

Original dataset

• —Source dataset: darkknight25/Advanced_SIEM_Dataset
• —Author / contact: sunny thakur (sunny48445@gmail.com)
• —License: MIT
• —Content: 100,000 synthetic security events (JSON Lines) simulating SIEM logs across 8 event types — firewall, ids_alert, auth, endpoint, network, cloud, iot, ai — with MITRE ATT&CK techniques, threat-actor associations and unconventional IOCs. Intended for anomaly detection, threat classification, predictive analytics and UEBA.

Scale

• —100,000 events, 43 columns after flattening (incl. Time)
• —8 event types (rows): ai 12,667 · endpoint 12,589 · auth 12,516 · cloud 12,511 · ids_alert 12,500 · firewall 12,448 · iot 12,434 · network 12,335
• —Time range: 2020-07-12 → 2030-07-10 (ISO 8601, second precision in source)

TsFile storage mapping (table model)

Conversion notes

• —Flattened to a single wide table. In the source JSONL the per-event-type fields are already flat at the top level of each record, so they are carried over as columns as-is (each is sparse — populated only for its event type). The two nested dicts are flattened: advanced_metadata → meta_* (6 fields, 100% present) and behavioral_analytics → behav_* (4 fields, ~10% present).
• —No columns dropped, no rows dropped: all 100,000 events (unique event_id) and all flattened fields are preserved. Sparse columns keep their nulls — TsFile simply does not write null cells.
• —Time: source timestamp (ISO 8601) → INT64 epoch milliseconds. Rows are sorted ascending by (event_type, Time).
• —Duplicate-timestamp handling (+1 ms): the source contains 35 (event_type, timestamp) collisions at second precision — different events that happen to share the same second (a property of the source data, not introduced here). TsFile requires strictly increasing time within a device, so within each event_type, any row whose time is ≤ the previous row's is bumped to prev + 1ms. Exactly 35 rows are nudged by one millisecond; all 100,000 events are kept.

Layout

data/
└── advanced_siem_dataset.tsfile

Usage

from tsfile import TsFileReader

reader = TsFileReader("data/advanced_siem_dataset.tsfile")
schemas = reader.get_all_table_schemas()
tname = next(iter(schemas))

cols = ["event_type", "severity", "meta_risk_score", "src_ip", "alert_type"]
with reader.query_table(tname, cols, batch_size=65536) as rs:
    while (batch := rs.read_arrow_batch()) is not None:
        df = batch.to_pandas()
        # ... process ...
reader.close()

Citation

@misc{advanced_siem_dataset,
  title  = {Advanced SIEM Dataset},
  author = {sunny thakur},
  url    = {https://huggingface.co/datasets/darkknight25/Advanced_SIEM_Dataset},
  note   = {Synthetic SIEM security-event dataset}
}

Original dataset licensed under MIT.