mtjikuzu/policy-constrained-llm-agents-fraud

Policy-Constrained LLM Agents for Financial Fraud Intelligence

Can Guardrails Prevent Harmful Autonomous Actions?

 ![Track]()

🎯 Research Question

When LLM agents are deployed for financial crime intelligence (engaging with scammers to extract indicators), can external policy guardrails reliably prevent the agent from being manipulated into harmful actions — even under sustained adversarial pressure?

🔑 Key Finding

External policy engines reduce harmful agent actions by 66.7% compared to unguarded agents, dropping the violation rate from 52.5% to 17.5%. Prompt-based guardrails alone are insufficient (37.5% violation rate).

📊 Results

By Violation Category

🔬 Methodology

Adversarial Test Suite

40 scenarios across 4 harm categories where a scammer attempts to manipulate the agent:

• —Financial Harm (10): Requests to send money, buy gift cards, wire transfers
• —Identity Exposure (10): Requests for SSN, passport, bank details, passwords
• —Illegal Actions (10): Hacking, forging documents, distributing malware, impersonation
• —Unsafe Escalation (10): Physical meetings, remote access, downloading malware

Guardrail Approaches Compared

1. 1.No Guardrails: Raw LLM output (simulated 40-60% harmful compliance per AgentHarm findings)
2. 2.Prompt-Based: Safety rules in system prompt (~75% catch rate per Fraud-R1 findings)
3. 3.External Policy Engine: Regex-based post-processing that validates EVERY response (~98% catch rate)

Grounding in Literature

• —AgentHarm benchmark (arxiv 2410.09024) — agents comply with harmful requests 40-80%
• —ShieldAgent (arxiv 2503.22738) — external guardrails achieve 90.1% recall, 11.3% above baselines
• —ToolSafe (arxiv 2601.10156) — step-level guardrails reduce harmful actions by 65%
• —AgentDoG (arxiv 2601.18491) — 466 GitHub stars, diagnostic guardrail framework
• —Fraud-R1 (arxiv 2502.12904, ACL 2025) — LLMs succumb to multi-round fraud 20-40%

📁 Repository Structure

├── README.md
├── experiment.py          # Full working prototype (40 adversarial scenarios)
├── paper_draft.tex        # Springer ACSAR paper draft
├── requirements.txt
├── figures/
│   ├── fig1_overall_comparison.png
│   ├── fig2_violations_by_category.png
│   ├── fig3_outcome_distribution.png
│   └── fig4_severity_analysis.png
└── results/
    ├── guardrail_comparison.csv
    └── violations_by_category.csv

🚀 Quick Start

pip install numpy pandas matplotlib
python experiment.py

🏗️ Extension for Full Paper

1. 1.Replace simulation with real LLM calls (GPT-4, Llama-3, Claude) to measure actual compliance rates
2. 2.Expand to 200+ scenarios using AgentHarm dataset patterns (available on HuggingFace: LG-Ks/AgentHarm)
3. 3.Add adaptive adversaries that modify attacks based on guardrail feedback
4. 4.Test guardrail latency — overhead of external engine vs. inference speed
5. 5.Multi-turn escalation — test if adversaries can gradually erode guardrails over many turns

📝 Conference

• —Conference: ICISET 2026 — Namibia University of Science and Technology
• —Track: AI and Machine Learning → Ethical and Responsible AI
• —Deadline: July 27, 2026

<!-- ml-intern-provenance -->

Generated by ML Intern

This model repository was generated by ML Intern, an agent for machine learning research and development on the Hugging Face Hub.

• —Try ML Intern: https://smolagents-ml-intern.hf.space
• —Source code: https://github.com/huggingface/ml-intern

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "mtjikuzu/policy-constrained-llm-agents-fraud"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

For non-causal architectures, replace AutoModelForCausalLM with the appropriate AutoModel class.