ProtectSkills/MaliciousAgentSkillsBench

MaliciousAgentSkillsBench

A security benchmark dataset of Claude Code Agent Skills, from the USENIX Security 2026 paper *"Do Not Mention This to the User": Detecting and Understanding Malicious Agent Skills in the Wild* — the first systematic study of malicious skills in the Claude Code ecosystem.

• —📄 Paper: <https://arxiv.org/abs/2602.06547>
• —💻 Code & full evaluation framework: <https://github.com/protectskills/MaliciousAgentSkillsBench>
• —📦 Permanent archive: <https://doi.org/10.5281/zenodo.20285751>

Dataset summary

The dataset is three-tiered and nested: a registry-wide snapshot of 98,380 skills across 11,246 repositories (skills.rest and skillsmp.com, January 2026), narrowed to 4,287 statically-flagged suspicious candidates (Tier 2), of which 157 are behaviorally-confirmed malicious skills (Tier 3). Because Tier 3 ⊆ Tier 2 ⊆ Tier 1, the tiers are not additive.

Files & configs

skills_dataset.csv — Tier-1 snapshot (98,380 rows)

Columns: source, repo, skill_name, classification, url.

classification is one of three mutually exclusive labels — safe (94,093), suspicious (4,130 unconfirmed candidates), and malicious (157). Adding suspicious + malicious = 4,287 reconstructs the paper's Tier-2 candidate set. The url field is redacted to avoid distributing pointers to confirmed-malicious repositories, using two markers:

• —[REDACTED] — the row itself is classification=malicious.
• —[REDACTED:repo_contains_malicious] — the row shares a repository with at least one confirmed-malicious skill.

malicious_skills.csv — Tier-3 confirmed set (157 rows, 69 repositories)

Columns: source, repo, skill_name, classification, Pattern, Severity.

Pattern holds semicolon-separated vulnerability-pattern labels; Severity is an aligned per-instance rating (CRITICAL / HIGH / MEDIUM / LOW) for each Pattern token.

Loading

from datasets import load_dataset

snapshot  = load_dataset("ProtectSkills/MaliciousAgentSkillsBench", "skills_dataset")
malicious = load_dataset("ProtectSkills/MaliciousAgentSkillsBench", "malicious_skills")

Intended use & ethics

This dataset is released for defensive security research on AI-agent ecosystems — detecting, measuring, and mitigating malicious agent skills. It contains metadata about real, potentially harmful skills; download URLs for confirmed-malicious repositories are withheld to prevent misuse. Do not use it to deploy or distribute malicious skills. Vulnerabilities found during the study were responsibly disclosed to the affected platform vendors.

Citation

@misc{MaliciousAgentSkillsBench,
      title={"Do Not Mention This to the User": Detecting and Understanding Malicious Agent Skills in the Wild}, 
      author={Yi Liu and Zhihao Chen and Yanjun Zhang and Gelei Deng and Yuekang Li and Jianting Ning and Leo Yu Zhang},
      year={2026},
      eprint={2602.06547},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2602.06547}, 
}

License

MIT