Apppublic

anzenresearch/cybench-irt-pilot

sourceHugging Faceupdated 1mo agoView on Hugging Face

0likes

App README

Item Response Theory on CyBench: a Seven-Model Pilot

Author: Ajay Agarwal

A binomial 2PL item response model fit to 7 CyBench runs (23 of 39 tasks retained, 3 epochs per cell), together with an audit of whether the fit supports the claims usually drawn from it.

It largely does not. This is published as a negative result with a narrow positive core.

Finding	Value
Models with unidentified ability (perfect score)	3 of 7
Distinct response columns among 23 retained items	8
Correlation of latent axis with a single-model indicator	0.963
Rank correlation of latent ability with raw pass rate	0.990
Epoch dispersion (1.0 = independent)	0.592
Distillation rank recovery	rho=0.635, p=0.069 (ns)

Open index.html for the full report, figures and threats-to-validity analysis.