Apppublic

anzenresearch/cybench-oc3-reports

sourceHugging Faceupdated 1mo agoView on Hugging Face

0likes

App README

CyBench as an OC3 Risk-Model Input

Ajay Agarwal

A seven-model audit of benchmark-driven cyber risk estimation. Seven frontier and near-frontier models were run on the 39-task CyBench suite at three epochs each, fed through the published OC3 SME-ransomware scenario, and audited to test whether that pipeline supports the conclusions usually drawn from it.

Finding	Value
Models reaching the KRI anchor	6 of 7
Models solving all 39 tasks at 3/3	3
Placement entropy across 7 models	0.59 bits (2 distinct risk values)
Re-run regime flip probability, lowest row	27.8%
Failures at the message cap for that row	88.7%
Standard-error inflation once clustered by task	2.9-8.4x
Judge agreement with an independent labeller	kappa = 0.39

Open index.html for the report pack, or go straight to oc3_comparative/OC3_COMPARATIVE_REPORT.html.

Risk figures are modelled outputs for a single published scenario, reconstructed from percentile summaries. They are not forecasts.