bijaye/precinct6-cybersecurity

WitFoo Precinct6 Cybersecurity Dataset

Overview

A large-scale, labeled cybersecurity dataset derived from production Security Operations Center (SOC) data processed by WitFoo Precinct version 6.x. The dataset contains 2.07 million sanitized security events (signal logs) and provenance graphs (10,442 incident graphs with 30,092 nodes and 595,618 edges) from real enterprise network monitoring across multiple organizations.

Available in two sizes:

• —`witfoo/precinct6-cybersecurity` — 2M signals (this dataset)
• —`witfoo/precinct6-cybersecurity-100m` — 114M signals (same methodology, larger extraction)

Generate your own: WitFoo Precinct 6.x customers can create datasets from their own data using the open-source pipeline: `witfoo/dataset-from-precinct6`

This dataset is designed to support research in:

• —Provenance graph-based intrusion detection (KnowHow, NodLink, and similar systems)
• —AI-driven cyber defense simulation (CybORG and MARL-based defense policy training)
• —Security alert classification (malicious vs. suspicious vs. benign event labeling)
• —Attack lifecycle analysis using MITRE ATT&CK framework mappings
• —Detection rule evaluation using WitFoo's 261 lead detection rules

Quick Start

from datasets import load_dataset

# Load flat signal logs (2.07M rows, Parquet)
signals = load_dataset("witfoo/precinct6-cybersecurity", "signals", split="train")

# Find malicious events
malicious = signals.filter(lambda x: x["label_binary"] == "malicious")

# Find suspicious events (matched detection rules but not in confirmed incidents)
suspicious = signals.filter(lambda x: x["label_binary"] == "suspicious")

# Query by product/vendor
cisco_events = signals.filter(lambda x: x["vendor_name"] == "Cisco")

# Load provenance graph
nodes = load_dataset("witfoo/precinct6-cybersecurity", "graph_nodes", split="train")
edges = load_dataset("witfoo/precinct6-cybersecurity", "graph_edges", split="train")

# Load full incident graphs (10K incidents with embedded artifacts, leads, and MITRE mappings)
incidents = load_dataset("witfoo/precinct6-cybersecurity", "incidents", split="train")

SQL queries (HuggingFace dataset viewer):

-- Malicious events from confirmed incidents
SELECT * FROM signals WHERE label_binary='malicious' LIMIT 100;

-- Suspicious events that triggered detection rules
SELECT * FROM signals WHERE label_binary='suspicious' LIMIT 100;

-- Events with matched detection rules
SELECT matched_rules, set_roles, product_name FROM signals
WHERE matched_rules != '[]' LIMIT 100;

-- Count by label tier
SELECT label_binary, COUNT(*) FROM signals GROUP BY label_binary;

Dataset Configurations

signals — Flat Security Signal Logs

Tabular format ideal for ML classification, anomaly detection, and feature engineering. Each row is a sanitized security event from production network monitoring.

graph_nodes — Provenance Graph Nodes

Nodes in the provenance graph representing network entities observed in security monitoring.

graph_edges — Provenance Graph Edges

Directed edges representing security events and relationships between entities.

incidents — Full Incident Graphs

Complete incident records as produced by WitFoo Precinct's threat detection engine. Each incident is a self-contained provenance graph capturing a correlated chain of suspicious or malicious activity. This is the richest configuration — each record contains embedded nodes, edges, leads (the triggering artifacts with full raw messages), and framework mappings.

Top-level fields:

Incident leads (leads.{uuid}):

Additional Files

• —`graph/graph.graphml` — Full provenance graph in GraphML format for NetworkX, Gephi, or graph databases
• —`graph/graph.json` — NetworkX node-link JSON format
• —`reference/lead_rules_catalog.json` — Complete catalog of 261 WitFoo lead detection rules, 158 security products, 106 classification sets, and 216 stream-to-product mappings

Data Provenance

Production Origin

This dataset was generated from production security operations data collected by WitFoo Precinct 6.x, a Security Orchestration, Automation, and Response (SOAR) platform. The data originates from real enterprise networks monitored by WitFoo's SOC platform, covering 5 organizations across diverse industry sectors.

Data collection period: July–August 2024

Processing pipeline:

1. 1.Security events were ingested by WitFoo Precinct 6.x from production network monitoring tools via syslog, API connectors, and agent-based collection
2. 2.Events were parsed by WitFoo's signal processing pipeline using field extractors specific to each product/vendor
3. 3.Events were correlated into incidents by WitFoo's automated threat detection and incident analysis engine using 261 lead detection rules
4. 4.Incidents were scored with WitFoo's suspicion scoring algorithm and mapped to security frameworks (MITRE ATT&CK, D3FEND, NIST, CIS, PCI, etc.)
5. 5.Raw signal data and incident graphs were extracted from WitFoo's Cassandra database
6. 6.All data was sanitized through a comprehensive 4-layer PII removal pipeline (see Sanitization)
7. 7.Labels were derived from WitFoo's incident analysis and lead rule matching (see Labeling)

Source Products (Stream Names)

The dataset contains security events from 158 security products across 70+ vendors, reflecting real enterprise SOC deployments. The complete product catalog is included in reference/lead_rules_catalog.json.

Key products by category:

Stream names in the dataset (top streams by volume):

WitFoo Precinct 6.x

WitFoo Precinct is a SOAR (Security Orchestration, Automation, and Response) platform that ingests security telemetry from diverse sources, parses events using vendor-specific field extractors, correlates events into incidents using automated threat detection rules, and maps findings to security frameworks.

• —Signal Processing: Multi-stage pipeline with field extraction, normalization, and enrichment
• —Lead Detection Rules: 261 rules that define what makes a security event suspicious enough to create an incident lead. Rules match on stream name, message type, action, severity, and vendor-specific event codes. Each rule assigns classification sets that define the role of source and target entities (e.g., Exploiting Host → Exploiting Target)
• —Incident Correlation: Automated grouping of related signals into incident graphs with nodes (hosts, credentials, actors) and edges (connections, communications)
• —Framework Mapping: Events and incidents are mapped to MITRE ATT&CK, MITRE D3FEND, NIST 800-53, NIST CSF, CIS Controls, PCI DSS, ISO 27001, SOC 2, and CMMC frameworks
• —Suspicion Scoring: Proprietary scoring algorithm that assigns suspicion levels to nodes and incidents based on observed behavior patterns

WitFoo Classification Sets

WitFoo classifies entities in incidents using 106 classification sets that define the role each entity plays in an attack. These are exposed in the set_roles column and in incident node data:

Labeling Methodology

Three-Tier Labels (malicious / suspicious / benign)

Labels are derived from two sources: WitFoo Precinct's incident analysis and lead detection rule matching.

• —`malicious`: The event was embedded as a lead (triggering artifact) inside one or more confirmed incidents. These events were identified by WitFoo's detection engine as part of attack patterns, correlated with other suspicious signals, and assigned to an incident with a suspicion score and modus operandi. The full artifact data, including raw messages, is extracted directly from the incident lead objects.

• —`suspicious`: The event matched one or more of WitFoo's 261 lead detection rules (e.g., "ASA Deny", "Windows Failed Login Attempt", "Blocked Action", "CrowdStrike Detection") but did not appear in a confirmed incident. These events represent security-relevant activity flagged by detection logic.

• —`benign`: The event did not match any lead detection rules and does not appear in any incident.

Label Distribution

2M Dataset (`witfoo/precinct6-cybersecurity`):

114M Dataset (`witfoo/precinct6-cybersecurity-100m`):

The imbalanced distribution reflects the reality of production SOC environments where the vast majority of events are benign, consistent with published IDS datasets (DARPA TC, LANL).

Lead Detection Rules

The matched_rules column contains JSON arrays of rule descriptions matched for each event. The complete rule catalog is in reference/lead_rules_catalog.json. Example rules:

MITRE ATT&CK Mappings

Attack technique and tactic labels are derived from WitFoo's framework mapping of incident patterns. The lifecycle_stage field maps events to the APT kill chain:

1. 1.initial-compromise — Initial access to the network
2. 2.establish-foothold — Execution and establishing persistence
3. 3.escalate-privilege — Privilege escalation attempts
4. 4.internal-reconnaissance — Discovery and internal scanning
5. 5.move-laterally — Lateral movement between hosts
6. 6.maintain-persistence — Command & control and persistence
7. 7.complete-mission — Data theft, exfiltration, or impact
8. 8.policy-violation — Policy violations (non-attack)

Incident Modus Operandi

Sanitization Methodology

All customer-identifying information has been removed through a comprehensive, iterative four-layer sanitization pipeline. Quality was prioritized over processing speed — the dataset underwent multiple full re-sanitization cycles until convergence (near-zero new PII discoveries per cycle). The sanitization pipeline is open source under the Apache 2.0 license.

Layer 1: Structured Field Sanitization with Multi-Pattern Sweep

Known data fields are sanitized based on their semantic meaning using deterministic replacement rules. IP addresses are replaced with reserved documentation ranges (RFC 5737 for public IPs, HMAC-based remapping for private IPs that preserves subnet relationships). Hostnames, usernames, organization names, email addresses, Windows Security Identifiers, AWS account numbers, and credentials are each replaced with consistent sequential tokens (e.g., HOST-0001, USER-0001, ORG-0001). All replacements are consistent — the same original value always maps to the same sanitized token across every record, preserving network relationships and graph topology essential for security research.

After field-level sanitization, every record is swept using an Aho-Corasick multi-pattern matching automaton built from the full registry of over 300,000 known PII values. This catches PII that appears in unexpected contexts such as concatenated strings, cross-field references, and embedded data structures. Product identifiers (vendor names, event types, pipeline names) are explicitly protected from this sweep to preserve the security-relevant metadata researchers need.

Layer 2: Format-Specific Log Message Parsing

Raw security log messages come in diverse vendor-specific formats. Eight specialized parsers handle the major formats: Cisco ASA syslog, Microsoft Windows Security Event XML, Elastic WinLogBeat JSON, AWS CloudTrail, Palo Alto Networks, VMware vCenter, DNS resolution logs, and a comprehensive generic fallback parser. Each parser understands the exact structure of its format and sanitizes PII within structured fields like XML elements, nested JSON objects, and CSV columns — contexts where simple pattern matching would be unreliable.

Layer 3: Machine Learning Residual Detection

After rule-based sanitization, machine learning models scan a stratified random sample of sanitized records for residual PII that pattern-based approaches may miss. Two complementary models are used: Microsoft Presidio (powered by a spaCy natural language processing model) for entity recognition of persons, organizations, IP addresses, and email addresses; and a BERT-based Named Entity Recognition model for an independent second opinion on person, organization, and location entities. New discoveries are added to the PII registry and trigger a full re-sanitization pass across all records.

Layer 4: Large Language Model Contextual Review

A stratified random sample of sanitized records is reviewed by Anthropic's Claude for contextual PII detection. The model is prompted to identify subtle PII that statistical pattern matching and NER models commonly miss: organization names or abbreviations embedded in log messages, internal hostnames that reveal organizational structure, employee names in file paths or service descriptions, Active Directory group names, and geographic identifiers tied to specific offices or data centers. Findings trigger additional registry updates and re-sanitization.

Iterative Convergence

The four layers run in iterative cycles. PII discovered by the ML and AI layers in one cycle is added to the pattern-matching registry, ensuring it is caught automatically by Layer 1 in all subsequent cycles across the complete dataset — not just in the sampled records. Cycles repeat until the ML and AI layers find near-zero new discoveries, indicating convergence.

What Is Preserved

The sanitization preserves all security-relevant information needed for research:

• —Timestamps — Event timing, dwell time, and lateral movement sequences
• —Port numbers — Protocol behavior signals
• —Protocol types — TCP/UDP/ICMP classification
• —Severity levels — Event priority and criticality
• —Vendor event codes — Cisco ASA codes, Windows Event IDs, AWS API names
• —Action types — Block, permit, logon, logoff, file access
• —MITRE ATT&CK / D3FEND mappings — Framework technique and tactic IDs
• —Graph topology — Node relationships and connection patterns (via consistent IP/hostname replacement)
• —Product/stream identifiers — Which security tool generated the event (explicitly protected from sanitization)
• —Lead rule match results — Which detection rules matched each event

Intended Uses

Primary Use Cases

1. 1.Provenance graph-based intrusion detection research — Evaluate and benchmark graph-based IDS approaches (KnowHow, NodLink) on production-derived data
2. 2.AI cyber defense simulation — Train and evaluate reinforcement learning defense policies in CybORG and similar simulators
3. 3.Security alert classification — Build and evaluate ML models for three-tier (malicious/suspicious/benign) event classification
4. 4.Attack lifecycle analysis — Study attack progression patterns mapped to MITRE ATT&CK
5. 5.Detection rule evaluation — Analyze effectiveness of 261 lead detection rules across diverse security products

Research Context

This dataset was produced in collaboration with the University of Canterbury (New Zealand) Computer Science and Software Engineering department for two research projects:

• —AI Cyber-Security Battle Simulator — Improving CybORG with realistic IDS observations, graph-based defense policies, and AI-driven attacker modeling
• —Intrusion Detection based on Provenance Graphs — Evaluating reproducibility and generalizability of KnowHow and NodLink detection methods

Limitations

• —Label imbalance: Production SOC data is inherently imbalanced (~92–99% benign). Sampling strategies may be needed for balanced training.
• —Temporal scope: Data covers July–August 2024, a limited time window
• —Organization diversity: Data from 5 organizations, each with different security tool deployments
• —Sanitization trade-offs: Some log message detail is reduced by PII replacement, particularly in free-text fields
• —Label derivation: Labels depend on WitFoo's automated detection and 261 rules; some attacks may be unlabeled (false negatives) and some benign events may be incorrectly flagged
• —Incident coverage: The same 10,442 incidents appear in both the 2M and 114M datasets since incidents are stored separately from signal data

Ethical Considerations

• —All customer-identifying information has been removed through the 4-layer sanitization process with ~302,000 PII mappings
• —The dataset does not contain personally identifiable information (PII) of any individual
• —IP addresses, hostnames, usernames, and organization names have been replaced with consistent synthetic tokens
• —The dataset should be used for defensive security research only

Citation

@dataset{witfoo_precinct6_2025,
  title={WitFoo Precinct6 Cybersecurity Dataset: Labeled Provenance Graphs and Signal Logs from Production SOC Operations},
  author={WitFoo, Inc.},
  year={2025},
  url={https://huggingface.co/datasets/witfoo/precinct6-cybersecurity},
  license={Apache-2.0}
}

License

This dataset is released under the Apache License 2.0.