Datasetpublic

AgentSkillPrivacy/SkillLeakBench

SkillLeakBench A credential-leakage benchmark for LLM agent skills, from the ASE 2026 paper How Your Credentials Are Leaked by LLM Agent Skills: An Empirical Study. 📄 Paper: https://arxiv.org/abs/2604.03070 💻 Code & detection pipeline: https://github.com/AgentSkillsPrivacy/SkillLeakBench 📦 Archive: https://doi.org/10.5281/zenodo.19367969 Dataset summary We collected 170,226 skills from SkillsMP and analyzed a 17,022-skill sample with static secret extraction… See the full description on the dataset page: https://huggingface.co/datasets/AgentSkillPrivacy/SkillLeakBench.

sourceHugging Facemitupdated 3mo agoView on Hugging Face

0likes67downloads

Dataset Card

SkillLeakBench

A credential-leakage benchmark for LLM agent skills, from the ASE 2026 paper *How Your Credentials Are Leaked by LLM Agent Skills: An Empirical Study*.

—📄 Paper: <https://arxiv.org/abs/2604.03070>
—💻 Code & detection pipeline: <https://github.com/AgentSkillsPrivacy/SkillLeakBench>
—📦 Archive: <https://doi.org/10.5281/zenodo.19367969>

Dataset summary

We collected 170,226 skills from SkillsMP and analyzed a 17,022-skill sample with static secret extraction, dynamic sandbox testing, and manual review. This release contains the 520 affected skills (437 vulnerable + 83 malicious) and their 1,708 security issues, across 10 leakage patterns (4 vulnerability + 6 malicious), plus disclosure and popularity tables. The release is de-identified (developer usernames removed) and contains no live credential values.

Classification	Skills	Issues
Vulnerable	437	1,371
Malicious	83	337
Total	520	1,708

Files & configs

`skills_dataset.csv` — affected skills (520 rows)

Per-skill records. Columns: source, skill_name, classification (vulnerable / malicious), patterns (semicolon-separated), issue_count, severity.

`issues.csv` — security issues (1,708 rows)

One row per issue. Columns: skill_id, skill_name, classification, pattern_id, academic_code, pattern, severity.

`remediation_summary.csv` — disclosure outcomes (3 rows)

Columns: classification, total, resolved, remaining.

`popularity_hardcoded_repos.csv` — repository popularity (37 rows)

Name-free popularity of hardcoded-credential repositories. Columns: repo_status, stars, forks.

Leakage patterns. Vulnerability (4): Information Exposure, Hardcoded Credentials, Insecure Storage, Artifact Leakage. Malicious (6): Remote Exploitation, Defense Evasion, Credential Compromise, Data Exfiltration, Resource Hijacking, Persistence.

Loading

python

from datasets import load_dataset

affected    = load_dataset("AgentSkillPrivacy/SkillLeakBench", "affected_skills")
issues      = load_dataset("AgentSkillPrivacy/SkillLeakBench", "issues")
remediation = load_dataset("AgentSkillPrivacy/SkillLeakBench", "remediation")
popularity  = load_dataset("AgentSkillPrivacy/SkillLeakBench", "popularity")

Intended use & ethics

Released for defensive security research on agent skills. The data is de-identified and excludes live credential values; vulnerabilities were responsibly disclosed to the platform.

Citation

bibtex

@inproceedings{skillleakbench2026,
  title     = {How Your Credentials Are Leaked by {LLM} Agent Skills: An Empirical Study},
  author    = {Chen, Zhihao and Zhang, Ying and Liu, Yi and Deng, Gelei and Li, Yuekang and Zhang, Yanjun and Ning, Jianting and Zhang, Leo and Ma, Lei and Li, Zhiqiang},
  booktitle = {Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE)},
  year      = {2026}
}

License

MIT