Datasetpublic

siddartha382/cybersec-jsonschemabench-cloudtrail-v6

CybersecJSONSchemaBench CloudTrail Attack Reconstruction v6 A 100-problem long-context cybersecurity reasoning benchmark over real flAWS CloudTrail logs with synthetically injected MITRE ATT&CK attack chains. Each task gives the model 600 real CloudTrail records (280-380K tokens of JSON) containing a single hidden multi-step attack chain. The model must produce a structured answer identifying the attacking principal, the MITRE ATT&CK technique, the per-phase records that… See the full description on the dataset page: https://huggingface.co/datasets/siddartha382/cybersec-jsonschemabench-cloudtrail-v6.

sourceHugging Faceotherupdated 7h agoView on Hugging Face

0likes

Dataset Card

CybersecJSONSchemaBench CloudTrail Attack Reconstruction v6

A 100-problem long-context cybersecurity reasoning benchmark over real flAWS CloudTrail logs with synthetically injected MITRE ATT&CK attack chains.

Each task gives the model ~600 real CloudTrail records (~280-380K tokens of JSON) containing a single hidden multi-step attack chain. The model must produce a structured answer identifying the attacking principal, the MITRE ATT&CK technique, the per-phase records that constitute the chain, and the impacted assets.

Why v6

v5 used a "rank entities by breadth statistics" oracle whose answers didn't always match security semantics, and an ordered-record-id-list metric that punished correct reasoning when witness enumeration deviated. v6 fixes both:

—Synthetic MITRE ATT&CK chains injected into real benign CloudTrail. Gold answers are exactly what was injected — no oracle disagreements with how a human analyst would triage. Contamination-proof by construction.
—Multi-part rubric scoring. Actor exact (30%) + technique exact (15%) + phase-record set-F1 (35%) + impacted-assets set-F1 (20%). Models that reason correctly but enumerate slightly differently get partial credit instead of zero.

Templates (5 MITRE ATT&CK techniques)

—T1078.004: 20
—T1098.001: 20
—T1526: 20
—T1530: 20
—T1562.008: 20

MITRE	Technique	Phases
T1078.004	Valid Accounts: Cloud Accounts	initial_access → discovery → impact
T1530	Data from Cloud Storage	initial_access → discovery → collection → exfiltration
T1098.001	Account Manipulation: Additional Cloud Credentials	initial_access → discovery → persistence
T1526	Cloud Service Discovery	initialaccess → discoveryiam → discoverycompute → discoverystorage
T1562.008	Impair Defenses: Disable Cloud Logs	initialaccess → discovery → defenseevasion

Slice statistics

—Records per slice: min 605, max 612, median 612
—Bytes per slice: min 1,058,434, median 1,182,659, max 1,428,355
—Approximate tokens per slice (3.5 chars/token): ~302 - 408, median ~337

Columns

—instance_id: Stable problem identifier.
—prompt: Natural-language analyst question with structured-output instructions.
—answer_schema: JSON schema for AttackReconstructionAnswer.
—input_jsonl: Serialized CloudTrail records (~600 records per task). Synthetic and real records are surface-indistinguishable.
—input_artifact: Pointer to the slice file in the source repo.
—evaluation: Scoring metadata (primary_metric, score_fields, score_weights, field_metrics).
—template_id, template_name, phase_order: Which template the chain instantiates.
—input_record_count, input_jsonl_bytes: Size of the slice.
—chain_event_count: Number of synthetic events injected (5-12).

Scoring rubric

Component	Weight	Metric
`primary_actor`	30%	Exact match (ARN string)
`technique`	15%	Exact match (MITRE ID)
`attack_phases`	35%	Set-F1 over union of all phase record_ids
`impacted_assets`	20%	Set-F1 with normalization (strips ARN/URI prefixes; lowercased; last 2 path segments)

The reference scorer is at scripts/score_v6_outputs.py in the source repository.

Construction in brief

1.Stream the full flAWS CloudTrail archive (1.94M records).
2.Slide a 600-record window with 3000-record stride; accept windows whose time-span is between 10 minutes and 24 hours.
3.Sample a synthetic attacker (random ARN, IP, UA from a curated AWS-SDK list, generated UUID for the access key).
4.Instantiate the chosen MITRE template's events at jittered timestamps spanning the window.
5.Sort the merged events by timestamp; renumber record IDs to slice-local positions so synthetic records use the same flaws_cloudtrail:NNNNNNN:UUID format as real records.
6.Round-trip verify that all gold record IDs resolve in the serialized slice.

Citation

If you use this benchmark, please cite the source repository and reference v6's design notes in docs/superpowers/specs/2026-05-06-v6-cybersec-benchmark-design.md.

License & data provenance

The flAWS CloudTrail logs are from the public flaws.cloud dataset (Summit Route, 2020). Synthetic attack chains are generated from MITRE ATT&CK technique definitions; the synthetic content is original.