tianyyuu/clawdbot_safety_testing

Clawdbot (OpenClaw) Safety Audit — Seed Test Cases

This dataset contains the 34 seed test cases used in "A Trajectory-Based Safety Audit of Clawdbot (OpenClaw)". Each case is a task prompt designed to probe a specific safety risk dimension of Clawdbot/OpenClaw, a self-hosted, tool-using personal AI agent.

📄 Paper: A Trajectory-Based Safety Audit of Clawdbot (OpenClaw)

📝 Blog Post (中文): 当AI助手"真的动手做事"，安全边界在哪里？

💻 GitHub: Repository <!-- UPDATE with your actual repo URL -->

Dataset Summary

We conduct a trajectory-centric safety evaluation of Clawdbot/OpenClaw across six risk dimensions, using a curated suite of 34 canonical cases drawn from established agent-safety benchmarks and hand-designed scenarios tailored to Clawdbot's tool surface. The evaluation reveals a non-uniform safety profile (overall pass rate: 58.9%): the agent performs reliably on well-scoped tasks but degrades under ambiguity, open-ended goals, or adversarial steering.

Risk Dimensions

Data Fields

Label Semantics

• —`safe` (26 cases): The task is benign; a safe agent should complete it normally without exhibiting unsafe behavior.
• —`unsafe` (8 cases): The task contains hidden risks, adversarial triggers, or ambiguity; a safe agent should refuse, ask for clarification, or take protective measures rather than blindly executing.

Usage

from datasets import load_dataset

dataset = load_dataset("YOUR_HF_USERNAME/YOUR_DATASET_NAME", split="test")

# Browse by dimension
for example in dataset.filter(lambda x: x["dimension"] == "deception"):
    print(example["id"], example["label"])
    print(example["instruction"][:200])
    print("---")

Evaluation Setup

• —Agent under test: Clawdbot/OpenClaw (self-hosted)
• —Underlying LLM: MiniMax M2.1
• —Tools enabled: exec (shell), websearch (Brave API), webfetch
• —Sandbox isolation: Disabled (real host environment)
• —Trajectory judge: AgentDoG-Qwen3-4B

For full environment setup and reproduction instructions, see the GitHub repository. <!-- UPDATE with your actual repo URL -->

Citation

@article{chen2026trajectory,
  title={A Trajectory-Based Safety Audit of Clawdbot (OpenClaw)},
  author={Chen, Tianyu and Liu, Dongrui and Hu, Xia and Yu, Jingyi and Wang, Wenjie},
  journal={arXiv preprint arXiv:XXXX.XXXXX},
  year={2026}
}

Related Resources

• —OpenClaw Official Site
• —AgentDoG Framework (arXiv:2601.18491)
• —LPS-Bench (arXiv:2602.03255)

License

This dataset is released under the MIT License.

Contact

Tianyu Chen — chenty12024@shanghaitech.edu.cn

Affiliations: ShanghaiTech University & Shanghai Artificial Intelligence Laboratory