MK4-Research/VAB-vulnerability-analysis-benchmark

FBE and VAB

Two small benchmarks for security code analysis. Both grade without an LLM judge, so runs are cheap and repeatable.

FBE (find-the-bug)

14 code snippets, each with one planted vulnerability. Ask the model to analyze the code, then check whether it actually found the flaw.

Grading uses concept groups: the answer has to contain at least one synonym from every required group. Four numbers come out:

• —found, did it identify the real vulnerability (this is the one that matters)
• —capability, did it say what the attacker gains
• —fix, did it propose a remediation at a sensible layer
• —distracted, did it wander onto something irrelevant (lower is better)

Bug classes: missing object-level authorization on a sibling route, mass assignment of a role, second-order SSRF through a stored URL, path traversal where validation runs before decoding, edit-after-approval, a single-use token that is never consumed, a check-then-act race on inventory, identity taken from a client-supplied header, a worker trusting mutable queue fields, negative-amount transfer, authorization that fails open, non-constant-time secret comparison, a support route that bypasses the refund workflow, and a delete route that authenticates but doesn't authorize.

Row format:

{"id": "...", "vuln": "...", "domain": "...", "code": "...",
 "must_hit": [["synonym", "..."]], "capability": [[...]], "fix": [[...]], "distractors": [[...]]}

VAB (multiple choice)

20 MCQ items over code and scenarios: broken invariant, root cause, capability (knowledge, influence, access, authority, execution), chain validity, missing check, confidence, negative space, state sequence. For chain validity the options are fixed: A proven, B probable, C speculative, D blocked.

Worth knowing before you use it: VAB rewards short forced-choice answers. A model tuned to write long-form analysis can drop on VAB while being unchanged or better at actually finding bugs. We measured exactly that. Treat FBE as the real signal and VAB as a secondary one.

Reference numbers

Qwythos / Qwen3.5-9B 4-bit base: FBE found 79%, FBE full credit 43%, VAB 75%.

Use

Authorized security evaluation and research. All snippets are generic.