achinta3/cybersec-jsonschemabench-cloudtrail-natural-hard-v4

CybersecJSONSchemaBench CloudTrail Natural Hard v4

This is a 100-problem natural-prompt long-context cybersecurity reasoning subset built from the full flAWS CloudTrail corpus.

Each row contains a natural analyst-style question, a large CloudTrail JSONL context, and the JSON schema the answer must match. Gold answers are deterministic hidden-oracle results over the serialized slice and are not included in this public export.

Families

• —actor_recon_to_change: 20
• —apigateway_mapping_chain: 8
• —credential_consequence_chain: 15
• —cross_service_value_bridge: 4
• —ec2_unsuccessful_expansion: 28
• —s3_asset_pressure: 10
• —shared_client_divergence: 15

Columns

• —instance_id: Stable problem identifier.
• —prompt: Natural benchmark-style analyst question.
• —answer_schema: JSON schema for the expected answer.
• —input_jsonl: Serialized CloudTrail records for the problem context.
• —evaluation: Scoring metadata.
• —reasoning_family: Hidden-oracle family name.
• —difficulty_tags: Intended reasoning requirements.
• —candidate_count: Number of competing candidates under the hidden oracle.
• —distractor_count: Candidate-related records not in the gold answer.
• —primary_margin_over_runner_up: Dominance margin used to control ambiguity.
• —input_jsonl_bytes: Serialized context size.