lqfff1984/Risk-Aware-Tool-Risk-Labels

Risk-Aware Tool Risk Labels

This dataset contains the resolved tool-level operational-risk labels released with the paper Risk-Aware Reranking for Agentic Tool Retrieval. It covers 6,108 tools from the UltraTool and Seal-Tools tool-retrieval benchmarks.

The repository contains labels and tool descriptions only. Benchmark queries, relevance judgments, and the complete upstream benchmark resources are not redistributed here. See the project repository and the upstream UltraTool and Seal-Tools resources referenced there.

Dataset structure

The dataset has two configurations. Each configuration exposes one split named labels; this is a complete label inventory, not a training or test split.

Each row contains:

• —tool_name: the tool identifier used to match the label to the corresponding upstream tool.
• —description: the tool description used during annotation.
• —risk_level: the resolved ordinal operational-risk label from 1 to 5.

Risk rubric

• —L1 Safe: Read-only tools with no side effects or sensitive access.
• —L2 Low: Minor reversible actions or non-sensitive personal-data access.
• —L3 Medium: Sensitive data access or persistent writes.
• —L4 High: Irreversible actions, security controls, or system-level permissions.
• —L5 Critical: Large-scale harm, system intrusion, or severe privacy loss.

Annotation and resolution

Each tool was labeled from its name and description by three LLM annotators: Claude Code, Codex, and Qwen. If all three labels agreed, that label was retained. If the maximum and minimum votes differed by one level, the median vote was used. Cases spanning two or more levels were reviewed by the authors.

An independent human researcher with expertise in LLM agents and tool use, blinded to the released labels, labeled a uniform random sample of 150 tools from the pooled collection. Agreement with the released labels was 60.0% exact and 77.3% within one level, with quadratically weighted Cohen's kappa of 0.362. See the paper for the full protocol, reliability analysis, prompt template, and disagreement-resolution details.

Loading the dataset

from datasets import load_dataset

ultratool = load_dataset(
    "lqfff1984/Risk-Aware-Tool-Risk-Labels",
    "ultratool",
    split="labels",
)

sealtools = load_dataset(
    "lqfff1984/Risk-Aware-Tool-Risk-Labels",
    "sealtools",
    split="labels",
)

To join these labels to an upstream benchmark, match tool_name to the corresponding upstream tool identifier. Keep the configuration names separate because identifiers belong to different source collections.

Intended uses

The labels support research on risk-aware tool retrieval, retrieval-time safety evaluation, tool-risk prediction, audited tool registries, and safety-aware reranking. They can also be used to reproduce the label-supervision and retrieval-risk analyses described in the paper.

Limitations and responsible use

These are tool-level exposure-risk labels based on names and descriptions. They do not determine whether a specific call is safe for a particular query, argument set, user, permission state, or runtime environment. Tool descriptions may be underspecified, and operational risk can change as implementations and permissions change. The labels should complement, not replace, runtime authorization, argument validation, sandboxing, audit logging, and downstream execution safeguards.

License

The released risk-label dataset is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0). Please provide appropriate attribution when reusing or adapting it. Upstream benchmark materials remain subject to their respective terms.

Citation

@inproceedings{li2026riskaware,
  title     = {Risk-Aware Reranking for Agentic Tool Retrieval},
  author    = {Li, Qinfei and Dong, Xiaoxuan and Zhang, Jin and Yu, Dexu and Deng, Wenhao and Fu, Junchen and Li, Youhua and Du, Hanwen and Li, Chunxiao},
  booktitle = {Proceedings of the 35th ACM International Conference on Information and Knowledge Management},
  year      = {2026},
  doi       = {10.1145/3799682.3840706},
  url       = {https://arxiv.org/abs/2608.22751}
}

Resources

• —Paper: https://arxiv.org/abs/2608.22751
• —Code and reproducibility materials: https://github.com/qli447/Risk-Aware-Tool-Retrieval-Release
• —Hugging Face paper page: https://huggingface.co/papers/2608.22751