PudiHaripriya/soc-env-analyst

OpenEnv: Enterprise SOC Analyst (Procedural Threat Triage)

🧠 Overview

This project introduces a high-fidelity Security Operations Center (SOC) simulation environment designed for evaluating and training autonomous agents in realistic cybersecurity workflows.

Unlike toy environments, this system models:

• —Multi-step incident response pipelines
• —Procedural reasoning under constraints
• —Noisy, adversarial log analysis
• —Strict compliance-driven decision making

The agent operates as a Tier-1 SOC Analyst, tasked with:

• —Detecting threats from raw SIEM logs
• —Investigating using limited tools
• —Classifying attacks via MITRE ATT&CK
• —Executing mitigation under strict SOP constraints

🎯 Why This Environment Matters

Real-world SOC operations are:

• —High-risk (incorrect actions can break production systems)
• —Procedural (strict runbooks must be followed)
• —Noisy (signals buried in irrelevant logs)
• —Resource-constrained (limited investigation bandwidth)

This environment captures all of the above, making it a meaningful benchmark for agentic reasoning, not just pattern recognition.

🏗️ Repository Structure

.
├── models.py                # Custom SOC Pydantic models (Actions/Observations)
├── client.py                # SOC-specific persistent WebSocket client
├── openenv.yaml             # Official environment specification
├── pyproject.toml           # Project metadata & UV dependencies
├── uv.lock                  # Deterministic dependency lock
├── inference.py             # Baseline agent for local evaluation
├── __init__.py              # Root exports
├── README.md                # Documentation & HF Metadata
└── server/
    ├── __init__.py          # Exports MyEnvironment
    ├── app.py               # FastAPI server (Official OpenEnv entry)
    ├── Dockerfile           # UV-based container definition for HF Spaces
    ├── soc_graders.py          # Procedural reward & SOP validation logic
    └── my_env_environment.py   # Core SOC simulation engine logic

⚙️ Core Environment Design

🔍 Observation Space (SOCObservation)

• —logs: Procedurally generated SIEM logs (multi-format, noisy)
• —tool_results: Threat intelligence responses
• —remaining_steps: Time constraint
• —api_budget_remaining: Limited investigation budget
• —system_message: Dynamic feedback + memory

🎮 Action Space (SOCAction)

• —QUERY_INTEL → Investigate IP reputation
• —CREATE_TICKET → Initiate mitigation workflow
• —BLOCK → Block external threat
• —ISOLATE_INTERNAL_HOST → Contain internal compromise
• —IGNORE → Dismiss benign activity

🔥 Key Features

1. Infinite Procedural Generation

• —Every episode generates:
• —new IPs
• —new attack patterns
• —new noise distribution 👉 Prevents memorization, enforces reasoning

2. Strict SOP Enforcement

• —Mandatory workflow:

QUERY_INTEL → CREATE_TICKET → MITIGATE

• —Violations → penalties 👉 Mirrors real SOC runbooks

3. MITRE ATT&CK Integration

Agents must classify attacks using:

• —Credential Access
• —Exfiltration
• —Impact
• —Initial Access
• —Discovery

4. Resource Constraints

• —Limited API calls (QUERY_INTEL)
• —Forces strategic decision-making

5. Zero-Day Mechanic

• —30% chance threat intel is WRONG 👉 Agent must trust logs over tools

6. Chaos Mode (Multi-Threat)

• —Multiple simultaneous attacks
• —Requires prioritization and sequential mitigation

7. Anti-Reward Hacking

• —Duplicate actions penalized
• —Invalid sequences blocked

8. Dynamic Feedback System

System message tracks:

• —Active tickets
• —Investigated IPs
• —Mitigations
• —Remaining threats

👉 Enables memory-based reasoning

🧪 Tasks & Difficulty Scaling

👉 Difficulty increases via noise, ambiguity, and multi-step reasoning

📊 Reward Design

• —Positive rewards for:
• —correct investigation
• —correct classification
• —correct mitigation

• —Strict Zero-Reward Terminations (Game Over) for:
• —Fatal SOP violations (e.g., blocking internal IPs)
• —Redundant actions (duplicate tickets)
• —Ignoring active zero-day threats 👉 Ensures agents cannot "reward hack" by spamming actions.

👉 Enables dense + meaningful learning signals

🤖 Baseline Agent

The baseline agent:

• —Uses an LLM (Qwen / OpenAI-compatible)
• —Operates under:
• —strict JSON schema
• —deterministic inference (temperature=0, seed=42)

Output Format

[START]
[STEP]
[END]

✔ Fully OpenEnv compliant ✔ Reproducible results

▶️ Running the Environment

1. Install dependencies

# Using UV (Recommended for OpenEnv)
uv sync
source .venv/bin/activate
# OR using pip
pip install -e .

2. Set API key

export HF_TOKEN="your_token_here"

3. Run inference

export SOC_TASK="hard"
python inference.py

🚀 Deployment (Hugging Face Spaces)

1. 1.Create a Docker Space
2. 2.Upload all files
3. 3.Wait until status = Running

Validate:

curl -fsSL https://raw.githubusercontent.com/neo-derek/open-env/main/scripts/validate-submission.sh | bash -s -- <your-space-url> .

🧠 Design Philosophy: Prompt-Driven SOP vs. Autonomous Exploration

The Critique

The strict SOP workflow may appear partially enforced through the system prompt, raising concerns about reduced reliance on pure environment-driven learning.

Our Perspective

In real-world SOC environments:

• —Agents cannot freely explore actions
• —Mistakes can cause production failures
• —Analysts follow strict procedural runbooks

Thus, the prompt represents operational prior knowledge, not shortcuts.

Environment Guarantees

• —Grader enforces SOP rules
• —Invalid actions are penalized
• —Correct sequencing is validated programmatically

👉 Environment correctness does not depend on prompt compliance

Future Direction

• —Introduce QUERY_SOP
• —Enable learning via interaction and feedback 👉 Move toward fully autonomous agent learning

🏆 Summary

✔ Realistic SOC simulation ✔ Strong OpenEnv compliance ✔ Multi-step reasoning challenges ✔ Deterministic evaluation ✔ Robust reward design

💡 Final Thought

This is not just an environment — it is a decision-making benchmark under real-world constraints.