Nobody05/cyber-security-100m

WitFoo Precinct6 Cybersecurity Dataset (large)

Overview

A large-scale, labeled cybersecurity dataset derived from production Security Operations Center (SOC) data processed by WitFoo Precinct version 6.x. This dataset contains 114,234,041 sanitized security events (signal logs) across 5 organizations and 12,361 incident provenance graphs (47,632 nodes, 32,086,552 edges).

Available in two sizes:

• —`witfoo/precinct6-cybersecurity` — 2.1M signals (smaller, faster to load; same incidents and graph methodology)
• —`witfoo/precinct6-cybersecurity-100m` — 114M signals (this dataset)

Generate your own: WitFoo Precinct 6.x customers can create datasets from their own data using the open-source pipeline: `witfoo/dataset-from-precinct6`

This dataset is designed to support research in:

• —Provenance graph-based intrusion detection (KnowHow, NodLink, and similar systems)
• —AI-driven cyber defense simulation (CybORG and MARL-based defense policy training)
• —Security alert classification (malicious vs. suspicious vs. benign event labeling)
• —Attack lifecycle analysis using MITRE ATT&CK framework mappings
• —Detection rule evaluation using WitFoo's 261 lead detection rules

Quick Start

from datasets import load_dataset

# Load flat signal logs (114M rows across 58 Parquet shards)
signals = load_dataset("witfoo/precinct6-cybersecurity-100m", "signals", split="train")

# Find analyst-confirmed malicious events
malicious_confirmed = signals.filter(
    lambda x: x["label_binary"] == "malicious" and x["disposition"] == "Disrupted"
)

# Find suspicious events that matched detection rules but are not in confirmed incidents
suspicious = signals.filter(lambda x: x["label_binary"] == "suspicious")

# Load provenance graph (47.6k nodes, 32M edges)
nodes = load_dataset("witfoo/precinct6-cybersecurity-100m", "graph_nodes", split="train")
edges = load_dataset("witfoo/precinct6-cybersecurity-100m", "graph_edges", split="train")

# Load full incident graphs (12,361 incidents with embedded artifacts, leads, frameworks)
incidents = load_dataset("witfoo/precinct6-cybersecurity-100m", "incidents", split="train")

Label Distribution

Disposition breakdown (the raw Precinct status, exposed for ground-truth stratification — see Ground Truth):

Signal Columns

Source Products

The dataset contains events from 158 security products across 70+ vendors. Complete catalog in reference/lead_rules_catalog.json.

Labeling Methodology

Three-tier labels derived from two sources:

• —`malicious` (74,064): Events embedded as leads inside confirmed incidents. Extracted directly from incident lead objects with suspicion scores, modus operandi, and MITRE mappings.
• —`suspicious` (616,605): Events matching WitFoo's 261 lead detection rules but not present in confirmed incidents.
• —`benign` (113,543,372): Events not matching any detection rules and not in any incident.

Ground Truth and Disposition

All labels in this dataset are derived from WitFoo Precinct's automated incident correlation engine — there is no independent, analyst-verified ground truth. Researchers should treat Precinct's analysis as a strong but imperfect oracle. The disposition column lets you assess label quality on a per-record basis:

The disposition_category column buckets these into four values for easier filtering: confirmed-malicious, false-positive, dismissed, automated. For experiments where ground-truth quality matters, restrict to records where disposition ∈ {Disrupted, Resolved} to compare against analyst-confirmed labels. This dataset has 41,391 records with `Disrupted` status, 10 with `Dismissed`, and the remainder with `Unprocessed` (no analyst review).

For benign and suspicious records, disposition is Unprocessed (no incident association). For malicious records, disposition reflects the parent incident's status at extraction time.

Scoring

The dataset exposes two related score fields:

• —`suspicion_score` (float, 0.0–1.0) — Precinct's proprietary suspicion score from the parent incident. Populated for malicious records; zero for benign and suspicious. (Previously this column was always 0 — that was a defaulting bug, now fixed.)

• —`label_confidence` (float, 0.0–1.0) — Confidence in the assigned label_binary tier. Computed deterministically from corroborating signal:

Note: label_confidence is not the probability the activity is malicious — it indicates how much corroborating evidence supports the assigned tier. Source: `src/precinct6_dataset/label.py`.

MITRE ATT&CK Mappings

Attack technique and tactic labels are derived from three sources, with deduplication:

1. 1.WitFoo set role names attached to the incident (e.g., C2 Server → TA0011 Command and Control, T1071 Application Layer Protocol)
2. 2.Modus operandi name on the incident (e.g., Ransomware → TA0001, TA0002, TA0040; T1486 Data Encrypted for Impact)
3. 3.Per-product framework data embedded in incident.nodes.products.frameworks (when present)

Tactic IDs use the standard MITRE ATT&CK Enterprise format (TA0001 through TA0043). Technique IDs are top-level techniques representing the most likely category for a given role. Researchers wanting precise per-event technique attribution should treat these as priors. Full mapping tables in `src/precinct6_dataset/mitre_mapping.py`.

Per-edge/per-node attribution in graph output: attack_tactics, attack_techniques, set_roles, lifecycle_stage, label_binary, label_confidence, suspicion_score, disposition are attached at the edge level in edges.jsonl and in per-incident GraphML files. Nodes in incidents.jsonl carry their own sets and products dicts with per-entity information.

Graph Data

Per-edge labels in edges.jsonl include attack_techniques, attack_tactics, set_roles, lifecycle_stage, disposition, mo_name, suspicion_score, incident_id (for INCIDENT_LINK edges).

Attack Reports

graph/attack_reports.jsonl contains one natural-language threat-hunting report per incident (12,361 reports). Each report is deterministically composed from structured incident metadata (modus operandi, set roles, lead descriptions, MITRE mappings, timestamps) and explicitly states that it reflects Precinct's automated correlation engine output, not an independent threat-hunting investigation.

Each record contains: incident_id, report_text, mo_name, suspicion_score, disposition, attack_techniques, attack_tactics, lead_count, set_role_names, matched_rules, products_observed, lifecycle_stage, and timing fields.

Researchers can audit exactly how each sentence is derived by reading `src/precinct6_dataset/attack_reports.py`.

Additional Files

• —`signals/signals-NNNNN.parquet` — 58 Parquet shards (~270 MB each), 2M rows per shard except final shard. Total ~15.7 GB.
• —`signals/metadata.json` — Per-shard summary, label/disposition distributions, top message types and streams.
• —`graph/nodes.jsonl` — 47,632 graph nodes (hosts + credentials derived from artifact src/dst/username).
• —`graph/edges.jsonl` — 32,086,552 graph edges, each with labels dict carrying MITRE, disposition, set_roles, etc.
• —`graph/incidents.jsonl` — Full incident records (12,361 lines, ~638 MB) with embedded nodes, edges, leads, and framework mappings.
• —`graph/incidents_graphml/{0-f}/{incident_id}.graphml` — 12,361 per-incident GraphML files, sharded into 16 subdirectories by first hex char of the incident UUID (HuggingFace caps directories at 10,000 files). Each file is small (KB-MB) and loadable in Gephi, NetworkX, igraph, or DGL. Ideal for graph-based research where loading the entire dataset is impractical.
• —`graph/attack_reports.jsonl` — Natural-language threat-hunting reports (12,361 reports, ~21 MB). See Attack Reports.
• —`reference/lead_rules_catalog.json` — Complete catalog of 261 WitFoo lead detection rules, 158 security products, 106 classification sets, and 216 stream-to-product mappings.

Note on the artifact-level graph: With 32M edges, the monolithic graph.graphml is intentionally not shipped at this scale — per-incident GraphMLs and the streaming edges.jsonl are the recommended entry points. Methodology and content fields are otherwise identical to the 2M version.

Sanitization

All customer-identifying information has been removed through a comprehensive 4-layer sanitization pipeline. The pipeline is open source under the Apache 2.0 license.

1. 1.Structured field sanitization + Aho-Corasick multi-pattern sweep — Known fields are replaced with deterministic tokens (IPs → RFC 5737 ranges, hostnames → HOST-NNNN, etc.). Every record is then swept using an Aho-Corasick automaton built from 97,000+ PII registry entries.
2. 2.Format-specific log message parsing — Eight specialized parsers handle Cisco ASA syslog, Windows Security Event XML, WinLogBeat JSON, AWS CloudTrail, Palo Alto Networks, VMware vCenter, DNS logs, and a generic fallback.
3. 3.Machine learning residual detection — Microsoft Presidio (spaCy NLP) and BERT NER scan for residual PII. New discoveries trigger full re-sanitization.
4. 4.Large language model contextual review — Claude reviews stratified samples for subtle PII. Findings trigger re-sanitization.

The four layers run in cycles. PII discovered by ML/AI in one cycle is caught automatically by Layer 1 in all subsequent cycles. Final PII registry: 97,046 unique mappings across 13 categories (IPs, hostnames, usernames, orgs, credentials, SIDs, emails, ARNs, etc.). All replacements are consistent — the same original value always maps to the same token, preserving graph topology.

Research Context

This dataset was produced in collaboration with the University of Canterbury (New Zealand) Computer Science and Software Engineering department for two research projects:

• —AI Cyber-Security Battle Simulator — Improving CybORG with realistic IDS observations, graph-based defense policies, and AI-driven attacker modeling
• —Intrusion Detection based on Provenance Graphs — Evaluating reproducibility and generalizability of KnowHow and NodLink detection methods

Limitations

• —Label imbalance: 99.40% benign reflects production SOC reality. Sampling strategies needed for balanced training.
• —Temporal scope: July–August 2024.
• —Ground truth: All labels derive from Precinct's automated correlation. Use the disposition column to stratify by analyst review level.
• —Shared incidents: The same 12,361 incidents appear in both the 2M and this dataset (incidents are stored separately from signal data; only the signal sample size differs).
• —Sanitization trade-offs: Some log message detail is reduced by PII replacement, particularly in free-text fields.

Citation

@dataset{witfoo_precinct6_100m_2026,
  title={WitFoo Precinct6 Cybersecurity Dataset (large)},
  author={WitFoo, Inc.},
  year={2026},
  url={https://huggingface.co/datasets/witfoo/precinct6-cybersecurity-100m},
  license={Apache-2.0}
}

License

Apache License 2.0