dkstr/drain4-spatiotemporal

drain4 — Drain3 reproduction + spatiotemporal log anomaly detection

A faithful reproduction of the Drain log template mining algorithm (He et al., ICSE 2018; stock implementation logpai/Drain3), extended with a spatiotemporal anomaly layer for Observability and Security anomaly detection on log pattern time series.

Layout

• —drain4/parser.py — Drain reproduction. Fixed-depth prefix tree (root → token-count → first tokens → leaf clusters), position-wise token similarity, numeric-token parametrization, template merging. Defaults match stock Drain3 exactly (depth=4, simth=0.4, maxchildren=100, no masking beyond numeric parametrization). Improvements over stock Drain3:
• —optional LCS similarity (position-independent, robust to shifted parameters)
• —stable template IDs (SHA-1 of the template), surviving restarts/re-mining
• —JSON persistence with no jsonpickle dependency
• —drain4/temporal.py — spatiotemporal engine. Per-template frequency series over time windows with four signal families and two severity profiles: | family | what it catches | strongest profile | |---|---|---| | counts | spikes / silences vs robust (median+MAD) baseline | observability | | novelty | first-ever templates, rare templates returning after long gaps | security | | correlation | templates appearing without their usual companions | both | | complexity | char entropy, repeated runs, injection-shaped tokens | security |
• —drain4/benchmark.py — template-parsing benchmark vs pip drain3 on LogHub 2k ground truth (HDFS, Zookeeper, BGL): majority-template accuracy, pairwise grouping precision/recall/F1, throughput.
• —drain4/eval_temporal.py — anomaly benchmark: injected anomalies (spike / silence / novel / injection) on the full Zookeeper stream vs a drain3+plain-z-score baseline, plus novelty AUC against real BGL alert labels.

Usage

from drain4 import TemplateMiner, SpatioTemporalEngine

miner = TemplateMiner(similarity="lcs")       # or "position" = stock Drain3 behaviour
engine = SpatioTemporalEngine(window_sec=300)
for ts, line in stream:
    cluster, change = miner.add_log_message(line)
    for ev in engine.process(ts, cluster.cluster_id, template=cluster.get_template(), raw=line):
        print(ev["window_start"], ev["template"], ev["signals"], ev["score_observability"], ev["score_security"])

Reproducing the benchmarks

pip install drain3
python drain4/benchmark.py                      # parsing axis (LogHub 2k ground truth)
python drain4/eval_temporal.py                  # temporal axis (Zookeeper injections + BGL labels)

Ground truth: logpai/loghub (Zookeeper 2k structured CSV verified; HDFS 2k; BGL 2k with per-line alert labels).