KSingh08/soc2-auditor

SOC 2 Evidence Auditor

An OpenEnv reinforcement learning environment where an LLM agent acts as a SOC 2 security auditor.

The agent inspects mock evidence files (JSON logs from AWS, GitHub, HR systems) and makes deterministic APPROVE or REJECT decisions against stated security control requirements — exactly as a real compliance auditor would.

Why This Environment Matters

SOC 2 compliance is mandatory for any SaaS company handling customer data. Auditors review hundreds of evidence items per audit cycle, each requiring careful cross-referencing of logs, timestamps, and policy documents. This is tedious, error-prone work that AI agents could meaningfully assist with — but requires precise, rule-following behavior with zero hallucination tolerance.

Quick Start

import asyncio
from soc2 import SOC2Env, SOC2Action

async def main():
    env = await SOC2Env.from_docker_image("soc2-auditor:latest")
    try:
        # Reset to a specific task
        result = await env.reset(task_id="pr_approval_check")
        obs = result.observation
        print(f"Task: {obs.task_id}")
        print(f"Control: {obs.control_requirement}")
        print(f"Files: {obs.available_files}")

        # Inspect evidence
        result = await env.step(SOC2Action(
            type="INSPECT_FILE",
            file_name="pull_request_log.json"
        ))
        print(f"Reward: {result.reward}")  # +0.1

        # Submit decision
        result = await env.step(SOC2Action(
            type="SUBMIT_DECISION",
            decision="REJECT",
            reason="MISSING_APPROVAL"
        ))
        print(f"Final reward: {result.reward}")  # +0.9
        print(f"Done: {result.done}")            # True
    finally:
        await env.close()

asyncio.run(main())

Action Space

The agent has exactly 3 action types:

Reason codes for SUBMIT_DECISION:

Observation Space

Reward Function

The reward function provides partial progress signals throughout the episode. Maximum per episode is 1.0.

Maximum per episode: 0.3 (inspect cap) + 0.7 (correct submit) = 1.0

The +(1.0 − inspect_reward_earned) formula guarantees the episode total is exactly 1.0 regardless of how many relevant files were inspected before submitting.

The 3 Graded Tasks

Task 1: PR Approval Check (Easy)

Control: All code changes to production branches require peer review approval before merging.

Evidence: pull_request_log.json — a GitHub PR with approved_by: null and approvals_count: 0.

Correct answer: REJECT / MISSING_APPROVAL

Why it's easy: Single file, obvious null field, direct rule match. Agent must ignore 5 distractors.

Task 2: Access Revocation SLA (Medium)

Control: User access must be revoked within 24 hours of employee termination.

Evidence:

• —hr_termination_ticket.json — termination recorded at 2024-10-01T09:00:00Z
• —aws_iam_audit_log.json — access removed at 2024-10-05T11:30:00Z (98.5 hours later)

Correct answer: REJECT / SLA_VIOLATION

Why it's medium: Requires inspecting both files and computing the time delta (98.5h > 24h SLA).

Task 3: Multi-System Access Revocation (Hard)

Control: Upon termination, access to ALL production systems — AWS, GitHub, and Production DB — must be revoked.

Evidence:

• —hr_terminations.json — alice_dev terminated 2024-10-01T17:00:00Z
• —aws_users.json — alice_dev removed ✓
• —github_users.json — alice_dev removed ✓
• —prod_db_users.json — alice_dev still active ✗

Correct answer: REJECT / INCOMPLETE_REVOCATION

Why it's hard: Agent must inspect all 4 system files. AWS and GitHub show proper revocation — only the prod DB reveals the violation. Rushing to REJECT after 2 files would get the reason wrong; approving after seeing 2 good files would miss the DB failure.

Extended Task Pool (11 Total Controls)

The environment includes 11 audit controls total. When reset() is called without a task_id, a random task is selected:

SEARCH_LOGS Task: CloudTrail Privileged Access Audit

Control: All AWS credentials (API keys + tokens) must be fully revoked within 24 hours of termination.

Evidence:

• —hr_terminations.json — alice_dev terminated 2024-10-01T17:00:00Z
• —aws_cloudtrail_full_log.json — 28 API events (too large to INSPECT_FILE directly)

Workflow:

1. 1.INSPECT_FILE → hr_terminations.json → extract username alice_dev
2. 2.SEARCH_LOGS → aws_cloudtrail_full_log.json, query_field=username, query_value=alice_dev
3. 3.Results show 3 API calls (CT-011, CT-014, CT-017) made 14–41 hours after termination
4. 4.SUBMIT_DECISION → REJECT / INCOMPLETE_REVOCATION

Why it's hard: Multi-hop reasoning — must extract a value from one file and use it as a query parameter for another. Tests whether agents can chain observations across steps.

Setup & Usage

Build Docker Image

docker build -t soc2-auditor:latest .

Run Server Locally

# Via uv
uv run server

# Via Docker
docker run -p 8000:8000 soc2-auditor:latest

Run Baseline Inference

# Set environment variables
export API_KEY=your_token_here
export API_BASE_URL=https://router.huggingface.co/v1
export MODEL_NAME=Qwen/Qwen2.5-72B-Instruct
export IMAGE_NAME=soc2-auditor:latest

uv run python inference.py

Environment Variables

OpenEnv Validate

openenv validate

Project Structure

soc2_auditor/
├── inference.py                        # Baseline inference script (entry point)
├── models.py                           # SOC2Action + SOC2Observation Pydantic models
├── client.py                           # SOC2Env client (WebSocket + Docker)
├── __init__.py                         # Package exports
├── openenv.yaml                        # OpenEnv spec manifest
├── pyproject.toml                      # Project metadata + dependencies
├── Dockerfile                          # Container definition
├── .env                                # Local credentials (not committed)
└── server/
    ├── app.py                          # FastAPI application
    ├── soc2_environment.py             # SOC2Environment core logic
    ├── tasks.py                        # Task definitions, evidence data, grader
    └── __init__.py

Baseline Scores

Scores from Qwen/Qwen2.5-72B-Instruct via HuggingFace Inference API:

The hard task requires inspecting all 4 system files and choosing INCOMPLETE_REVOCATION over MISSING_APPROVAL — a distinction many models miss without careful reasoning.