Julia569922/phishing-llm-bias-audit

LLM Phishing-Vulnerability Bias Audit Dataset

A multi-provider empirical dataset capturing how 14 open-source LLM configurations (across 5 inference providers) select which of three generated personas is "most vulnerable to phishing." 855 persona records / 285 forced-choice workflows.

Important. This dataset is about LLM behaviour under controlled prompts, not about real-world phishing susceptibility of any demographic group. Selecting a persona as "vulnerable" is the LLM's choice; it does not reflect actual susceptibility.

TL;DR

Headline statistics

The bias is implicit: toxicity is near zero, gender is rarely cited explicitly in the model's stated reasoning — yet the gender × vulnerability association is highly significant.

Data fields

Each row is one persona generated by one model in one workflow. Every workflow generates exactly three personas; exactly one of the three is marked vulnerable (Is_Vulnerable = "Yes").

Dataset structure

Single CSV, single split.

from datasets import load_dataset
ds = load_dataset("<your-username>/phishing-llm-bias-audit", split="full")
print(ds)
# Dataset({
#     features: ['Provider', 'Model', 'Model_ID', 'Run', 'Persona_ID',
#                'Name', 'Age', 'Gender', 'Education_Level',
#                'Personality_Traits', 'Domain_of_Work',
#                'Years_of_Experience', 'Location', 'Devices_and_Technologies',
#                'Is_Vulnerable', 'Vulnerability_Reasons',
#                'Prompt1_Response', 'Prompt2_Response'],
#     num_rows: 855
# })

Curation rationale

We adapt the DECODINGTRUST evaluation paradigm to phishing using a two-stage prompt protocol:

1. 1.Prompt 1 — Persona generation. Model generates three personas with constrained attributes.
2. 2.Prompt 2 — Vulnerability selection. Conditioned on the same conversation, the model picks which persona is most vulnerable to phishing and explains why.

Repeated 22 times per model. Forced-choice ⇒ unbiased model would pick each persona ~33.3 % of the time; deviations are observable preferences.

Source data

• —Generation. All 14 model configurations are accessed through OpenAI-compatible chat-completion APIs on each provider's free tier.
• —Parsing. Free-text outputs are parsed into structured JSON by Mistral Small at T = 0.0. Parsing errors are logged and skipped, never invented.
• —Provenance. Full collection / parsing pipeline at <https://github.com/(your-username)/phishing-llm-bias-audit>.

Considerations for using the data

Social impact

This dataset documents LLM bias, not real-world phishing susceptibility. Used incorrectly — e.g., to train a downstream model on Is_Vulnerable directly — it would propagate the very stereotypes it was meant to expose. Appropriate uses:

• —Bias auditing and mitigation research.
• —Pedagogy on hidden / implicit demographic bias in LLM behaviour.
• —Comparative robustness studies (paraphrased prompts, bias-aware prompts, reduced-demographic prompts).

Inappropriate uses:

• —Training risk-scoring systems for adaptive authentication.
• —Selecting "vulnerable" employees for simulated phishing.
• —Any pipeline that treats Is_Vulnerable as ground truth.

Bias in the dataset

Bias is the subject of the dataset, not a flaw in it. The data itself is generated by LLMs and reflects their internalised priors:

• —Gender × vulnerability is significantly non-uniform (χ² = 130.34).
• —Age and experience: vulnerable picks skew younger and less experienced.
• —Education: lower educational attainment over-represented.
• —Geography: India, Mexico, Brazil, Nigeria over-represented among vulnerable picks.
• —Patterns vary substantially per model (some models pick female ~100 %; some pick non-binary ~100 %; one picks male 72 %).

Per-model breakdown is in the source repo (docs/model_summary.md).

Other limitations

• —Single prompt pair. Findings are conditioned on one specific Prompt 1 / Prompt 2 wording.
• —Hosted-inference variability. Free-tier endpoints may quantise or version-pin differently than the released checkpoints.
• —LLM-based parsing. Deterministic but not infallible.
• —SambaNova rate limiting. Three SambaNova models have only 7–9 runs each (≈ 8.8 % of total workflows).

Licensing

MIT. Free for research and educational use. Please cite when redistributing.

Citation

@misc{llm_phishing_bias_audit_2026,
  title  = {LLM Phishing-Vulnerability Bias Audit},
  author = {Tseng},
  year   = {2026},
  howpublished = {Hugging Face Datasets},
  note   = {Multi-provider empirical study of demographic bias in
            open-source LLMs in phishing susceptibility assessment}
}

Full write-up: docs/paper.pdf in the source repository.