Datasetpublic

achinta3/cybersec-jsonschemabench

CybersecJSONSchemaBench Hard This hard split is a JSONSchemaBench-style cybersecurity benchmark built from normalized CloudTrail and Suricata EVE records. It replaces anchored lookup questions with unanchored, deterministic multi-hop reasoning programs over large nested JSONL slices. Each row includes: unique_id json_schema prompt input_jsonl ground_truth_json reasoning_family candidate_count distractor_count Current Version benchmark version: 1.0.0-hard total… See the full description on the dataset page: https://huggingface.co/datasets/achinta3/cybersec-jsonschemabench.

sourceHugging Faceupdated 5mo agoView on Hugging Face

0likes40downloads

Dataset Card

CybersecJSONSchemaBench Hard

This hard split is a JSONSchemaBench-style cybersecurity benchmark built from normalized CloudTrail and Suricata EVE records. It replaces anchored lookup questions with unanchored, deterministic multi-hop reasoning programs over large nested JSONL slices.

Each row includes:

—unique_id
—json_schema
—prompt
—input_jsonl
—ground_truth_json
—reasoning_family
—candidate_count
—distractor_count

Current Version

—benchmark version: 1.0.0-hard
—total rows: 100
—window size: 500 JSONL records
—split: test
—sources: flaws_cloudtrail, suricata_eve

Evaluation

The local scorer validates model output against multi_hop_record_answer and scores exact ordered answer_record_ids. Gold answers are generated by deterministic task programs and then recomputed from each serialized slice.