monish211205/incident-response-env

🚨 Incident Response OpenEnv

An OpenEnv environment where an AI agent acts as an on-call Site Reliability Engineer, diagnosing and resolving production incidents across a simulated microservices system.

Why This Environment?

Every company running production software faces incidents — services going down, latency spiking, cascading failures rippling through a dependency graph. Incident response is one of the most high-stakes, time-pressured reasoning tasks in software engineering.

This environment captures that task faithfully:

• —The agent reads real-looking monitoring dashboards (service metrics, PagerDuty-style alerts)
• —It runs diagnostic commands (inspecting log tails from failing services)
• —It must identify the root cause — not just the symptom — through a dependency graph
• —It applies the correct remediation: rollback a bad deployment, scale up an overloaded service, or restart one stuck in a crash loop
• —All under a ticking SLA clock

This is exactly what SRE teams at Google, Meta, Netflix, and Stripe do every day.

Environment Design

System Topology

Each episode presents a snapshot of a microservices system. Services have:

Action Space

The agent submits one action per step — mirroring real SRE operations:

Every action requires a rationale — the agent's reasoning. This is scored for specificity (mentioning service names, metrics, root-cause keywords).

Observation Space

{
  "incident_title": "[P1] Site-wide degradation — multiple services failing",
  "incident_severity": "P1",
  "step_count": 2,
  "max_steps": 10,
  "time_to_resolve_budget": 50,
  "services": [
    {
      "name": "auth-service",
      "status": "degraded",
      "error_rate": 0.30,
      "latency_p99_ms": 3200,
      "cpu_utilisation": 0.97,
      "memory_utilisation": 0.85,
      "recent_deployment": false,
      "dependencies": ["redis-cache"]
    }
  ],
  "active_alerts": [
    {
      "alert_id": "ALT-020",
      "service": "api-gateway",
      "severity": "P1",
      "message": "api-gateway: error rate 45% — multiple upstream failures"
    }
  ],
  "last_diagnostic": {
    "service": "auth-service",
    "log_tail": ["2026-03-27T16:00:05Z [ERROR] Token validation timeout — queue full"],
    "error_summary": "auth-service is CPU-saturated due to a 2x traffic spike...",
    "suggested_action": "SCALE_UP auth-service"
  },
  "action_history": [
    "Step 1: [INVESTIGATE] -> api-gateway | Starting at user-facing entry point..."
  ]
}

Tasks

Easy — Single Service Failure

Scenario: Payment service crashes immediately after a bad deployment. API Gateway begins returning 502s. Alerts fire on both services.

The agent must:

1. 1.Investigate the API Gateway (upstream errors point to payment-service)
2. 2.Investigate payment-service (crash logs show failed DB migration)
3. 3.Rollback payment-service
4. 4.Resolve

Medium — Cascading Failure (Multi-hop)

Scenario: Checkout latency exceeds 10 seconds. The root cause is inventory-service stuck in a crash loop — three hops from the user-facing alert.

The agent must trace: checkout → order → inventory.

Hard — Site-wide Degradation with Red Herrings

Scenario: Four services simultaneously degraded with P1 alerts firing. Root cause is auth-service CPU saturation from a 2x traffic spike — but:

• —notification-service had a recent deployment (red herring)
• —user-service has a worse error rate than auth-service
• —auth-service has a lower severity alert (P2) than the symptoms (P1)
• —The fix is scale_up — not restart or rollback

Reward Function

Reward is dense — partial progress signals fire every step.

Episode-end reward (90% of final score)

Per-step reward (10% of final score)

Penalties

API Reference

Baseline Scores

Local Setup

git clone https://huggingface.co/spaces/monish211205/incident-response-env
cd incident-response-env
pip install openenv-core fastapi "uvicorn[standard]" pydantic openai
uvicorn server.app:app --host 0.0.0.0 --port 7860 --reload

Open http://localhost:7860/docs for the interactive Swagger UI.

Run tests

pip install pytest pytest-asyncio
pytest tests/ -v
# Expected: 34 passed

Run baseline

python -m baseline.baseline

Project Structure

incident-response-env/
├── server/
│   ├── app.py           # FastAPI server (openenv-core create_app)
│   ├── environment.py   # IncidentResponseEnv — core environment class
│   ├── models.py        # Typed Pydantic models: Action, Observation, State
│   ├── scenarios.py     # Incident scenario definitions (easy/medium/hard)
│   ├── reward.py        # Dense multi-component reward function
│   └── graders.py       # Deterministic graders for all 3 tasks
├── baseline/
│   └── baseline.py      # LLM + rule-based baseline agent
├── tests/
│   └── test_environment.py  # 34-test suite
├── openenv.yaml         # OpenEnv metadata spec
├── Dockerfile           # Multi-stage production build
├── pyproject.toml       # Package configuration
└── README.md            # This file

License

Apache 2.0 — see LICENSE.