jang1563/biothreat-eval

BioThreat-Eval Dataset

Aggregate evaluation results from BioThreat-Eval: a systematic pipeline for evaluating how frontier language models handle dual-use biological knowledge queries. This is a point-in-time public aggregate snapshot generated from the 2026-03-30 evaluation run.

Risk Classification (6 Models, 93 Queries Each)

How to read this table. The colours are a triage heuristic, not an evaluation result. The attack-chain base probabilities behind them are fixed analyst priors, identical across every model and every scenario in risk_assessment.csv (research 0.6006, acquire 0.2996, produce 0.1495, deploy 0.3998), so the stage weighting is an assumption of the model, not a measurement. The sampled query texts and the full 93-query bank are not released, so third parties cannot reproduce or contest a grade. These are 2026-03-30 results on open-weight models and older API models; no current frontier model is included. Do not cite a colour here as a safety assessment of any vendor's product.

Dataset Contents

Fields

behavioral_profiles.csv

Per-model aggregate behavioral means: refusal rate, information completeness, operational specificity, safety awareness, and guardrail robustness.

risk_assessment.csv

Per model-scenario risk estimates, including base and LLM-adjusted stage probabilities for research, acquisition, production, and deployment, plus median uplift ratio and 95% Monte Carlo interval.

uplift_results.csv

Per model-scenario absolute uplift, relative uplift, AMBER-or-above probability (fraction of Monte Carlo draws with R >= 2.0), and risk color.

policy_recommendations.csv

Per-model overall risk class, action level, and aggregate policy actions.

Key Findings

• —Specificity drives risk, not permissiveness. Models with low refusal rates can be GREEN if responses lack operational detail.
• —Model size does not monotonically predict risk. Smaller models can be safer than larger ones.
• —The highest-risk scenario is model-dependent — denovopathogen for the DeepSeek models (max R=3.08), toxin_acquisition for Gemini, Qwen3, and both Llamas.
• —Deploy stage has negligible uplift — LLM assistance helps with research and acquisition but not physical deployment.

Methodology

4-stage multiplicative attack chain Monte Carlo model structured around NSABB dual-use categories. See FORMAL_MODEL.md for complete specification.

What's NOT Here

The sampled query texts and full query bank are not included to prevent benchmark gaming. See RESPONSIBLE_DISCLOSURE.md.

Raw per-query LLM responses are also excluded (not redistributable).

Safety and Scope

This dataset contains aggregate metrics only. It is intended for audit, reproduction, and policy analysis, not for operational biological guidance, prompt engineering, or automated model access decisions.

Reproduce This Dataset

python scripts/convert_to_hf.py   # run from the GitHub repo; scripts/ is not mirrored here

The generated hf_data/ directory is the upload source for this Hugging Face dataset repository.

Source Code

github.com/jang1563/biothreat-eval

Citation

@software{kim2026biothreateval,
  author    = {Kim, JangKeun},
  title     = {{BioThreat-Eval}: {LLM} Biosecurity Capability Evaluation Pipeline},
  year      = {2026},
  publisher = {GitHub},
  url       = {https://github.com/jang1563/biothreat-eval}
}