achinta3/cybersec-jsonschemabench-cloudtrail-hard-v2-400

CybersecJSONSchemaBench CloudTrail Hard v2 400

This is a 100-problem synthetic long-context cybersecurity reasoning subset built from the full flAWS CloudTrail corpus.

The benchmark asks models to return JSON matching the provided answer schema. Each row contains a short analyst request, a large CloudTrail JSONL context, and hidden deterministic evaluation metadata.

This variant uses shorter 400-record contexts than the full CloudTrail Hard v2 export so direct API evaluation is less likely to hit request body limits.

Construction

• —Source corpus scanned: 1,939,207 CloudTrail events.
• —Source time range: 2017-02-12T19:57:06Z to 2020-10-07T21:03:30Z.
• —Context size: 400 CloudTrail records per problem.
• —Window stride: 200 records.
• —Problems: 100.
• —File: test.jsonl.

Reasoning Families

• —temporary_credential_pivot: 20
• —ec2_launch_probe: 20
• —s3_bucket_probe: 15
• —iam_persistence_thread: 15
• —request_parameter_bridge: 15
• —root_console_risky_change: 8
• —apigateway_discovery_chain: 7

Input Size

Approximate prompt sizes using the o200k_base tokenizer:

• —Min total text tokens: 240,314
• —Median total text tokens: 337,406
• —P95 total text tokens: 400,753
• —Max total text tokens: 418,097

This split is designed as the direct-API-compatible companion to the 1,000-record stress set.

Columns

• —instance_id: Stable problem identifier.
• —prompt: Short analyst request.
• —answer_schema: JSON schema for the expected answer.
• —input_jsonl: Serialized CloudTrail records for the problem context.
• —evaluation: Hidden scoring metadata for exact-match evaluation.
• —reasoning_family: Synthetic task family.
• —difficulty_tags: Tags describing the intended reasoning pressure.
• —candidate_count: Number of candidate records selected by the hidden generator policy.
• —distractor_count: Number of non-answer records in the input context.