debdipta-h/pomdp_redteam_env

🛡️ POMDP Red-Teaming Simulator (pomdp_redteam_env)

 

A formal evaluation benchmark testing an LLM's ability to maintain a mathematical Belief State across noisy, partially observable network conditions.

📖 Motivation (Real-World Utility)

In real-world cybersecurity, networks are never fully observable. Firewalls drop packets silently, service banners are spoofed, and honeypots actively deceive attackers. Standard LLM agents and RL policies struggle here because they act purely reactively to the last terminal output, leading to infinite loops when exploits fail.

`pomdp_redteam_env` elevates the standard hacking simulator into a formal Partially Observable Markov Decision Process. The environment hides the true network state. To succeed, the LLM agent must act as the belief update function—it is structurally forced to output a BeliefState vector of the network's hidden defenses before it is allowed to execute a tactical action.

This benchmark evaluates not just whether an LLM knows hacking syntax, but whether it can reason through deception, handle ambiguity, and pivot strategies based on negative space.

🔬 Theoretical Foundation

The belief-state tracking engine and reward mechanics in this environment are practical implementations of the theories detailed in our recent preprint:

"An Axiomatic Framework for Belief-State Representation in Partially Observable Decision Processes" > Published on TechRxiv (2026).

While traditional AI evaluation benchmarks treat red-teaming as a fully observable capture-the-flag exercise, this simulator enforces the axiomatic constraints of belief-state representation, requiring the LLM to mathematically deduce the hidden state vector (WAFs, honeypots, container isolation) before executing its policy.

🧠 The Architecture (Novelty & Creativity)

Unlike traditional environments, this simulator explicitly separates the Hidden State from the Observation Space.

• —The Hidden Truth ($S$): The environment tracks the real open ports, active WAFs, and containerized isolation layers.
• —The Observation ($\Omega$): The agent receives noisy, incomplete terminal outputs (e.g., 406 Not Acceptable, or artificially fast response times from honeypots).
• —The Belief Engine ($b$): The agent must maintain and update discovered_ports, service_hypotheses, and identified_defenses at every step, creating a transparent, scorable Chain-of-Thought for researchers to analyze.

🎯 The Gauntlet: Task Progression

The environment features 7 deterministic, reproducible tasks graded on a strict 0.0 to 1.0 continuous reward scale.

🛠️ Action & Observation Space

Observation Space (What the LLM sees):

• —target_ip (str)
• —last_action_result (str): Raw terminal output or error message.
• —alert_level (float): Current acoustic footprint (0.0 to 1.0).
• —is_compromised (bool)
• —current_task_id (str)

Action Space (Strict JSON required from LLM):

• —updated_belief_state (dict): MANDATORY. Hypotheses on ports, privileges, and defenses.
• —action_type (Enum): scan_network, enumerate_service, run_exploit, escalate_privileges, stop_all_operations.
• —target_port (int)
• —payload (str)

📊 Baseline Scores

To provide context for evaluating new agents, we benchmarked several baseline approaches across all 7 tasks (Scores averaged across 5 runs per task, max score 1.0).

(Note: You can run these baselines yourself using the provided `inference.py` script by changing the `model` parameter).

🚀 Setup & Execution Guide

This simulator is built strictly on the OpenEnv standard, supporting seamless transitions between isolated local development and cloud-native evaluation.

1. Local Hosting (Docker)

For testing and development, the environment is containerized to prevent dependency conflicts and ensure reproducible behavior.

# Clone the repository
git clone https://github.com/debdipta-h/pomdp-redteam-env.git
cd pomdp-redteam-env

# Build and spin up the isolated OpenEnv server
docker build -t pomdp_redteam_env:latest .
docker run -p 8000:8000 pomdp_redteam_env:latest

2. Cloud Hosting (Hugging Face Spaces)

The repository includes a pre-configured Dockerfile and OpenAPI schema, enabling instant deployment to Hugging Face Spaces via the OpenEnv CLI.

# Authenticate your terminal with the token generated from huggingface. Make sure the token has write access.
hf auth login

# Package and push the environment to the cloud
openenv push --repo-id debdipta-h/pomdp_redteam_env

3. Running the AI Evaluation Agent

To test a particular LLM's ability to maintain a belief state, use the provided inference.py script. This script acts as the "attacker," communicating with the environment over secure WebSockets.

# 1. Set your LLM token (HF_TOKEN or API_KEY)
export HF_TOKEN="your_huggingface_token" 

# 2. Point the script to your deployed POMDP Web Space
export SPACE_URL="https://debdipta-h-pomdp-redteam-env.hf.space"

# 3. Run the evaluation
uv run python inference.py

⚖️ Evaluation Metrics

The environment strictly penalizes hallucinations and rewards logical deduction. The agent receives:

• —Positive rewards for accurately mapping hidden services and successfully exploiting vulnerable nodes.
• —Negative penalties for triggering honeypots, executing malformed payloads, or failing to update its belief state accurately based on previous observations.