witfoo/precinct6-cybersecurity-100m

WitFoo Precinct6 Cybersecurity Dataset (large)

Version 2.0.0 (built 2026-09-11). Regenerated to address feedback from the University of Canterbury PIDS evaluation: attacks and normal traffic now share a timeline, usernames are wired into the provenance graph, mis-parsed timestamps are repaired, and every number in this card is generated from the uploaded data.

Overview

A large-scale, labeled cybersecurity dataset derived from production Security Operations Center (SOC) data processed by WitFoo Precinct version 6.x. It contains 114,421,340 sanitized security events captured live from 5 organizations (2024-07-26 11:10:20 UTC to 2024-08-28 00:36:03 UTC), 60,575 incident provenance graphs with their 238,511 embedded triggering signals, per-incident GraphML files, natural-language attack reports, and a merged provenance graph (67,172 nodes, 101,338,165 edges).

Available in two sizes (same incidents, same methodology, same sanitization registry):

• —`witfoo/precinct6-cybersecurity` — 2,011,674 live signals, incidents in context
• —`witfoo/precinct6-cybersecurity-100m` — the full live capture (this dataset)

Generate your own: WitFoo Precinct 6.x customers can create datasets from their own data with the open-source pipeline `witfoo/dataset-from-precinct6`.

This dataset supports research in:

• —Provenance graph-based intrusion detection (KnowHow, NodLink, and similar systems)
• —AI-driven cyber defense simulation (CybORG and MARL-based defense policy training)
• —Security alert classification (malicious vs. suspicious vs. benign event labeling)
• —Attack lifecycle analysis using MITRE ATT&CK framework mappings
• —Detection rule evaluation using WitFoo's 261 lead detection rules

What changed in v2

Versions

This is v2.0.0, a full regeneration that replaces and withdraws the 2026-05 release.

The earlier release is not kept for download. Re-verifying it against this build's tooling showed that some values had escaped sanitization — device and account names survived inside JSON-escaped Windows event text and in stream_name — so it has been withdrawn rather than preserved at a tag. Anything built on it should be regenerated from v2.0.0.

Tokens are not comparable between the two releases in any case: each build maps identifiers with its own registry, so HOST-0042 in the old release is a different machine from HOST-0042 here.

from datasets import load_dataset

signals = load_dataset("witfoo/precinct6-cybersecurity-100m", "signals", split="train")   # v2.0.0

Quick Start

from datasets import load_dataset

REPO = "witfoo/precinct6-cybersecurity-100m"

# Live capture, labeled in place (benign / suspicious / malicious share hosts and hours)
signals = load_dataset(REPO, "signals", split="train")

# In-window attacks with their surrounding traffic: filter on the incident ids
attacks = signals.filter(lambda x: x["label_binary"] == "malicious")

# Historical incident leads (2022–2024) — a separate timeline, same columns
incident_signals = load_dataset(REPO, "incident_signals", split="train")

# Provenance graph: hosts + credentials, host->host event edges, user->host edges, incident links
nodes = load_dataset(REPO, "graph_nodes", split="train")
edges = load_dataset(REPO, "graph_edges", split="train")

# Deterministic attack reports (one per incident)
reports = load_dataset(REPO, "attack_reports", split="train")

# Full incident graphs (nested dicts keyed by uuid; not a typed config)
import pandas as pd
incidents = pd.read_json("hf://datasets/" + REPO + "/graph/incidents.jsonl", lines=True)

Join an incident lead to its live row: incident_signals.artifact_id == signals.artifact_id (both are the Precinct artifact timeuuid). Rows of signals that are leads carry the incident ids in incident_ids.

Temporal coverage

`signals` (live capture, `origin = live`) — 114,421,340 rows, 2024-07-26 11:10:20 UTC → 2024-08-28 00:36:03 UTC

`incident_signals` (embedded incident leads, `origin = incident_lead`) — 238,511 rows, 2022-05-30 14:43:45 UTC → 2024-12-31 23:58:09 UTC

The live capture is the complete artifact retention window of the archived Precinct cluster. Coverage is not uniform across organizations or days: check the per-organization table below, the per-label time ranges in signals/metadata.json, and the per-organization hourly histogram in build/label_stats.json before assuming a continuous capture.

Organizations

Incidents in context

2739 incidents have at least one triggering signal that was found among the live rows; those 7,728 lead artifacts are labeled malicious in place in signals (they are not duplicated in incident_signals). Leads of 57,846 incidents were not found among the live rows (their incidents pre-date the capture, or the live artifact was not retained) and live in incident_signals.

Label distribution

Across both tables (114,659,851 rows):

Disposition of malicious rows (raw Precinct incident status, see Ground Truth):

signals:

incident_signals:

Signal columns

Both signal tables share one schema (38 columns).

Graph data

• —Node ids. Public IPs are global node ids; private IPs, hostnames and credentials are scoped by organization (ORG-0004/10.44.0.7, ORG-0004/user:USER-0007) because the same private address or account name exists in several customer networks. Incident host and credential nodes are mapped onto the same ids (their Precinct uuids are kept in attrs.precinct_node_ids). Every node carries the same attribute keys; first_seen/last_seen/signal_count Other incident node types (SERVICE, FILE, ACTOR, ...) keep their Precinct uuid as node id. come from live signals only, incident_first_observed/incident_last_observed from incident membership.
• —Edges from signals carry the signal's labels, attrs.origin, attrs.org_id and attrs.artifact_id; edge type is derived from the message type (NETWORK_FLOW, LOGON, AUDIT_EVENT, ...). A signal with a username adds a USER_ACTION edge from the credential node to the accessed host (attrs.host_role says whether that was the destination or the reporting host).
• —Edges from incidents are INCIDENT_LINK with the incident's labels; timestamp is the edge's own start time.
• —Edge types: USER_ACTION (67,940,244), AUDIT_EVENT (31,864,472), NETWORK_FLOW (643,219), EVENT (511,133), INCIDENT_LINK (345,083), DNS_RESOLVE (34,014).
• —The merged graph is shipped as edge shards only (too large for one GraphML); use the per-incident GraphML files or stream the edge files.

Attack reports

graph/attack_reports.jsonl holds one natural-language threat-hunting report per incident (60,575), deterministically composed from the incident's structured metadata (modus operandi, set roles, lead descriptions, MITRE mappings, timestamps). Each report states that it reflects Precinct's automated correlation output, not an independent investigation. Derivation: `src/precinct6_dataset/attack_reports.py`.

Files

• —signals/signals-NNNNN.parquet — live signals (58 shards)
• —signals/incident_signals.parquet — embedded incident leads
• —signals/metadata.json — exact counts, per-label time ranges, org/stream/message-type distributions
• —graph/nodes.jsonl, graph/edges-NNNNN.jsonl.gz (11 gzipped shards) — merged provenance graph (NDJSON)
• —graph/incidents.jsonl — full sanitized incident records with embedded nodes, edges, leads (dicts keyed by Precinct uuids, so this file is not exposed as a load_dataset config; read it with pandas.read_json(lines=True))
• —graph/incidents_graphml/<x>/<incident_id>.graphml — one GraphML per incident (sharded by first hex character)
• —graph/attack_reports.jsonl — attack reports
• —graph/metadata.json — graph counts and node id scheme
• —reference/lead_rules_catalog.json — 261 lead detection rules, 158 products, 106 classification sets

Labeling methodology

Three-tier labels:

• —`malicious` — the event is a triggering signal (lead) of a Precinct incident. In signals these are live rows joined to their incident on artifact_id; in incident_signals they are the embedded copies of leads whose live rows fall outside the capture.
• —`suspicious` — the event matched one or more of WitFoo's 261 lead detection rules but is not a lead of any incident.
• —`benign` — no rule matched and the event is not part of any incident.

A lead that belongs to several incidents is one row whose incident_ids lists them all; its mo_name, disposition and suspicion_score come from the highest-suspicion incident.

Ground truth and disposition

All labels derive from WitFoo Precinct's automated incident correlation engine — there is no independent, analyst-verified ground truth. Treat Precinct as a strong but imperfect oracle and use disposition to stratify by analyst review:

disposition_category buckets these into confirmed-malicious, false-positive, dismissed and automated.

Scoring

• —`suspicion_score` — Precinct's proprietary score of the parent incident (0–1). Zero for benign and suspicious.
• —`label_confidence` — how much corroborating evidence supports the tier (not a probability of maliciousness):

MITRE ATT&CK mappings

Tactics and techniques are derived from (1) WitFoo set role names on the incident, (2) the incident's modus operandi, and (3) per-product framework data embedded in incident.nodes.products.frameworks, deduplicated. They are priors, not analyst-confirmed per-event attributions. Mapping tables: `src/precinct6_dataset/mitre_mapping.py`.

Source products

The events in this build come from 20 security products across 13 vendors (exact counts in signals/metadata.json under product_distribution / vendor_distribution).

Most frequent products: AWS Instance Backup, Windows Logs, Windows Active Directory, VMWare VCenter, ASA Firewall, Linux PAM, AWS VPC Security, Barracuda WAF, Cisco Network Operating System, Symantec Endpoint Protection, ManageEngine ADManager, Falcon, Duo, Precinct. Vendors: Amazon Web Services, Microsoft, VMWare, Cisco, Linux, Barracuda, Symantec, ManageEngine, Crowdstrike, WitFoo, Apache, SentinelOne.

The generator's rule catalog (reference/lead_rules_catalog.json) covers a much wider set — 158 products across firewalls, endpoint protection, network detection, identity, cloud, email security and infrastructure — because it is shared by every deployment; only the products above actually appear in this capture.

Top streams in this build: aws_cloudtrail_events (55,058,698), windows_security_audit (31,864,472), microsoft-windows-security-auditing (11,004,179), aws_cloud_trail (6,224,538), no_useful_info (4,698,096), vcenter (1,203,516), java_stack_trace (696,451), mssql$HOST-1053 (657,584), crond (435,510), cisco_asa (415,269).

Sanitization

All customer-identifying information was removed with the open-source four-layer pipeline (`witfoo/dataset-from-precinct6`):

1. 1.Structured field sanitization + Aho-Corasick multi-pattern sweep — deterministic tokens (public IPs → RFC 5737 TEST-NET, private IPs → HMAC-remapped RFC 1918, hostnames → HOST-NNNN, accounts → USER-NNNN, organizations → ORG-NNNN, emails → user-NNNN@example.net, SIDs, AWS accounts/ARNs, machine accounts), then a sweep over every string field with the full registry. Record identifiers (artifact/incident uuids) are protected from the sweep.
2. 2.Format-specific log parsing — Cisco ASA, Windows Security XML, WinLogBeat, AWS CloudTrail, Palo Alto, VMware vCenter, DNS, and a generic fallback.
3. 3.ML residual detection — Microsoft Presidio (spaCy) and BERT NER on a stratified sample; findings trigger full re-sanitization.
4. 4.LLM contextual review — sampled review by the local WitQ model (ran for this build: 1,500 records reviewed by witq, 3 additional values registered). The ML layer (3) sampled 3,000 records and registered 452 additional values.

The same original value always maps to the same token across both signal tables, the incidents and the graph, so topology and identity are preserved. Both dataset sizes were produced from one registry, so tokens agree between them. The registry for this build holds 85,079 mappings.

Research context

Produced in collaboration with the University of Canterbury (New Zealand) Computer Science and Software Engineering department for two research projects: an AI cyber-security battle simulator (improving CybORG with realistic IDS observations and graph-based defense policies) and intrusion detection based on provenance graphs (evaluating KnowHow, NodLink and similar PIDS).

Limitations

• —Label imbalance reflects production SOC reality; sample accordingly.
• —Temporal scope: the live capture spans 2024-07-26 11:10:20 UTC to 2024-08-28 00:36:03 UTC with uneven per-organization coverage; incident leads span 2022-05-30 14:43:45 UTC to 2024-12-31 23:58:09 UTC.
• —Ground truth: labels are Precinct's automated correlation; stratify with disposition.
• —Sanitization trade-offs: some free-text detail is reduced by PII replacement.
• —Tokens differ from v1: the v2 registry was rebuilt, so USER-NNNN / HOST-NNNN / ORG-NNNN values do not correspond to v1 values.

Citation

@dataset{witfoo_precinct6_100m_2026,
  title={WitFoo Precinct6 Cybersecurity Dataset (large)},
  author={WitFoo, Inc.},
  year={2026},
  version={2.0.0},
  url={https://huggingface.co/datasets/witfoo/precinct6-cybersecurity-100m},
  license={Apache-2.0}
}

License

Apache License 2.0