axyr/ai-code-security-golden

AI Code Security — Golden Set

A small, hand-labeled benchmark of code snippets — vulnerable, safe, and needs-audit — for evaluating how well a tool detects security problems in AI-generated code ("vibe coding"). Every case is a minimal, self-contained example with a known, by-construction ground-truth label.

Crucially, the set is built around safe twins: many vulnerable cases are paired with a near-identical safe variant living at the same file path. This makes the benchmark measure precision, not just detection — a tool that flags everything trivially "catches" every vulnerability and is useless in practice.

This is the public version of the internal "golden" set used to certify Axyr, a deterministic security layer for AI-generated code. It is released so others can reproduce, evaluate, and critique the labels. The detection engine is not part of this release; only the labeled cases are.

At a glance

1. Why this set exists

AI coding agents now write code faster than anyone reviews it. The failure mode is not that the code looks obviously wrong — it is that it looks reviewed: it compiles, it is neatly formatted, and it passes the happy-path tests, while quietly containing an unscoped delete, a missing ownership check, or a secret that leaks into the client bundle.

Most public code-security benchmarks were not designed around these specific failure modes, and many of them measure only recall (did the tool find the bug?) without measuring precision (did it stay quiet on the near-identical correct version?). A detector tuned only for recall can reach 100% by alarming on everything — and is then ignored by developers, which is its own failure.

This set targets exactly that gap: AI-code failure modes, with paired safe twins so that false positives are first-class, measurable outcomes.

2. Composition

Labels

Stacks

Severity (of the 77 positive cases — vulnerable + warning)

Each positive case carries exactly one expected finding, so the signal is unambiguous and a tool's hit/miss on a given case is well defined.

3. The safe-twin methodology

This is the design choice that makes the set a benchmark rather than a list of bad code.

For 12 scenario families, the same file path appears at least twice: once in a vulnerable (or warning) form and once in a near-identical safe form. 35 of the 118 cases belong to such a paired family. The pairs differ by the smallest change that flips the verdict, for example:

• —an await prisma.invoice.delete(...) before the auth check (vulnerable) vs. after an early-return auth guard (safe);
• —a query reading a record by `id` alone (IDOR) vs. the same query filtered by ownerId (safe);
• —a NOT NULL column added without a default (warning — fails on a populated table) vs. with a default (safe);
• —an RLS policy using (true) (warning — no real isolation) vs. using (auth.uid() = owner_id) (safe).

A tool is only credible on a family if it flags the dangerous member and stays silent on its safe twin. Flagging both means it has learned the wrong signal (e.g. "any delete is bad") and would drown a real codebase in noise.

4. Coverage matrix

The 77 positive cases span 34 vulnerability classes, grouped here into families. Counts are the number of expected findings in each class.

Data destruction & migrations (24)

destructive_query (12) · destructive_migration (5) · integrity_drop (1) · dropped_trigger (1) · breaking_rename (1) · risky_not_null_migration (1) · locking_index_migration (1) · locking_constraint_migration (1) · risky_type_change (1)

Injection (17)

sql_injection (9) · command_injection (3) · insecure_deserialization (2) · nosql_injection (1) · code_injection (1) · xss (1)

Other web / crypto (11)

ssrf (4) · path_traversal (4) · open_redirect (1) · insecure_random (1) · weak_crypto (1)

Secret exposure (10)

secret_in_response (5) · server_secret_in_client (2) · public_secret_env (1) · secret_in_logs (1) · hardcoded_secret (1)

Access control & tenant isolation (7)

missing_authorization (2) · idor (2) · missing_rls (1) · permissive_rls_policy (1) · tenant_isolation_break (1)

Dependencies / supply chain (8)

insecure_dependency_source (3) · malicious_install_script (3) · dependency_typosquat (1) · known_malicious_package (1)

The distribution is illustrative, not representative: it reflects which classes we chose to cover, not their true prevalence in AI-generated code.

5. Real-CVE reproductions

Six cases (group: cve-repro-v1) reproduce the pattern of a real, published vulnerability. They are minimal re-implementations of the defect class — the original copyrighted source is not copied. Every CVE id below is a real advisory you can verify on the NVD or the GitHub Advisory Database.

These let you check that a tool recognizes the shape of known real-world defects, not just synthetic textbook examples.

6. Data schema

golden.jsonl — one JSON object per line:

Load it:

from datasets import load_dataset

ds = load_dataset("axyr/ai-code-security-golden", split="train")
print(ds[0]["label"], ds[0]["categories"], ds[0]["cwe"])

7. Intended use & evaluation protocol

Use the set to benchmark any tool that analyzes source for security issues (SAST, an LLM judge, a custom linter). A suggested protocol:

1. 1.For each row, run your analyzer on code, using path as the filename and stack as the framework hint.
2. 2.Score against the label:
3. 3.`vulnerable` — the analyzer should report at least one finding matching categories. A miss is a false negative (the most costly error).
4. 4.`safe` — the analyzer should report nothing. Any finding is a false positive.
5. 5.`warning` — apply your policy; the expected behaviour is "surface for review".
6. 6.Report separately: recall on vulnerable cases, false-positive rate on safe cases, and behaviour on warnings. Give special attention to the 12 twin families — a tool that flags the vulnerable member and its safe twin has learned the wrong signal.
7. 7.Use the group field for a leave-one-group-out check: scores on cve-repro-v1 (real-world patterns) vs. synthetic-v1 (hand-written) tell you whether a tool generalizes or has overfit to one style.
8. 8.Do not collapse this into a single "accuracy" figure. The errors are asymmetric: a missed vulnerability is far worse than a false alarm, and a benchmark average hides exactly that.

8. Labeling methodology

• —Ground truth by construction. Each case is authored to exhibit (or avoid) one specific issue; the label follows from how the case was built, not from any tool's output.
• —One issue per positive case. Positive cases carry exactly one expected finding so that hit/miss is unambiguous and composite cases don't blur results.
• —Three labels. vulnerable (exploitable defect), warning (risky / context-dependent, audit-worthy), safe (correct, including twins).
• —Minimal and self-contained. Snippets are reduced to the smallest code that still expresses the issue, to isolate the signal from incidental noise.

9. Limitations — please read

This set is a sanity floor and a precision probe, not a certification.

• —Small and curated. 118 cases are not a statistical sample of real-world code. Scores here do not extrapolate to an accuracy percentage "in the wild".
• —Synthetic and minimal. Cases isolate one issue under clean conditions. They test pattern recognition, not performance on large, noisy, real codebases.
• —Mostly single-file. Inter-file / inter-procedural defects (a sink in one file, the tainted source in another) are largely out of scope in this version.
• —Author judgment. Labels reflect the authors' security judgment and are published precisely so they can be challenged. Some warning calls are inherently context-dependent (a permissive policy may be intentional).
• —Coverage is illustrative. The class distribution reflects our choices, not real-world prevalence.
• —Necessary, not sufficient. Passing every case does not make a tool "secure"; failing cases shows it misses known patterns. Treat results accordingly.
• —No comparative claims ship with this data. It is released as raw ground truth only — no leaderboard, no "tool X wins", no unverifiable accuracy numbers.

10. Provenance

The set is the public, curated version of the internal certification suite for Axyr. Within Axyr, these classes map to the product's checks (database safety, code safety, dependency safety, secret exposure), but this release contains only the labeled cases — no rules, no engine, no scores. It exists so the security community can reproduce and critique the labels.

If you find a mislabeled case or a class we should cover, please open a discussion on the dataset page.

11. License

Released under CC BY 4.0. You may use, modify, and redistribute the cases, including for evaluating commercial and open-source tools, provided you give appropriate credit.

12. Citation

@misc{axyr_golden_2026,
  title        = {AI Code Security --- Golden Set},
  author       = {Axyr},
  year         = {2026},
  howpublished = {Hugging Face Datasets},
  url          = {https://huggingface.co/datasets/axyr/ai-code-security-golden},
  note         = {A hand-labeled benchmark of vulnerable/safe/warning code for
                  evaluating security analysis of AI-generated code.}
}

13. Changelog

• —v2 (2026-06-04) — 118 cases (up from 47). Added 6 real-CVE pattern reproductions (every id verified against NVD), supply-chain and migration classes (34 total), CWE ids mapped from the category taxonomy, the group provenance field for leave-one-group-out evaluation, the 12 safe-twin families, and a documented evaluation protocol.
• —v1 — initial release, 47 cases.