dkstr/drain4-spatiotemporal

full-corpus v2 eval results

eval_v2: mining axis (drain4+param-split vs drain3 per source) + temporal axis (learn 0-2, calibrate on clean day 3, detect 7-13, family-aware sustain gates, counts-only baseline)

temporal v2: parent-keyed families, silence split from counts (presence-scored), sustained-mean spike signal, optional seasonal time-of-day baseline, episode grouping with hysteresis, O(1) rolling correlation

calibrated parser: param-split with masked sub-template rendering (template identity = parent), timestamp-shape exclusion, recursive sub-routing, sub-aware persistence

temporal v2: parent-keyed counts/novelty/corr (sub rollup), episode grouping with hysteresis, O(1) rolling co-occurrence, per-family thresholds, MAD=0 spike fix, lazy presence

parser v2: adaptive recursive param-split (cardinality-fitted freeze keys, timestamp-shape exclusion, line-unique ratio rule), sub-aware persistence, sub template IDs anchored to stable parent

synthgen v2: realistic sshd background (failed-pw share 18.6%->1.5%) + 6-IP brute-force pool

Upload results/synth_eval_results.json with huggingface_hub

eval_synth: absolute-time window indices (relative split made engine never train + labels never intersect); drain3 add_log_message returns dict

temporal: cap per-template entropy history at 1000 samples (unbounded insort was O(N^2) at 100M scale)

synthgen: source-major task ordering so web shards run in parallel (14-day wall ~15 min)

synthgen.py: validated generator (268 templates, 9 sources, 9 incidents, parquet shards) — smoke-tested at scale 0.005

synthgen.py: complete clean rewrite — registry, 9 sources, incidents, parquet shards, meta.json

Complete synthgen implementation (replaces truncated draft)

Synthetic multi-source log generator: ~500 templates, Zipf skew, diurnal+bursty arrivals, 9 injected incident types, parquet shards

benchmark results: eval_temporal_results.json

benchmark results: benchmark_results.json

Fix: eng.update=False before detect; m3.drain.clusters; rarity signal added to BGL axis

Perf + correctness: incremental novelty bookkeeping (first/last nonzero, gaps), trailing-window stats cap

Fix parquet path: default/partial-test (verified 74,273 rows)

Zookeeper source -> bolu61/loghub_2 test parquet (full 74k-line Zookeeper); two-window silence; raw content passed; deterministic injection placement

Fix infinite loop: advance current_window after closing (sub-agent found hang in _close_window loop)

Rewrite: robust_stats public, first-sighting fix, per-instance entropy history, bounded corr scan

README

package init

Clean rewrite: single inline-injection replay + drain3 z-score baseline + BGL novelty AUC

Spatiotemporal anomaly engine: 4 signal families + observability/security fusion

Fix tree_search wildcard fallback; simplify LRU dict

Temporal anomaly benchmark: injected anomalies on Zookeeper stream + BGL novelty AUC

Template-parsing benchmark: drain4 vs pip drain3 on LogHub 2k ground truth

Core Drain reproduction: fixed-depth prefix tree, position/LCS similarity, stable template IDs, JSON persistence

initial commit