tek-wizard/devops-incident-responder

DevOps Incident Responder

At 2:13 AM, login failures spike, dashboards turn red, and every minute of delay costs trust, revenue, and sleep.

A language model can already talk about incidents. It can summarize logs, explain what a rollback is, and suggest generic debugging steps.

That is not the same as behaving well inside a live outage.

DevOps Incident Responder is an OpenEnv benchmark for training and evaluating safer incident-response copilots in a realistic microservice production world. The agent must investigate noisy telemetry, maintain a world model across multiple steps, choose safe remediations in the right order, and verify that customer-facing systems actually recovered.

This submission is built for the Meta PyTorch OpenEnv Hackathon and is primarily aligned with:

• —Theme #3.1 World Modeling: professional, tool-facing, partially observable workflows
• —Theme #2 Long-Horizon Planning & Instruction Following: multi-step reasoning with delayed consequences and recovery from early mistakes

Links

• —Hugging Face Space: https://huggingface.co/spaces/tek-wizard/devops-incident-responder
• —Runtime URL: https://tek-wizard-devops-incident-responder.hf.space
• —Mini-blog: Blog.MD
• —Main training notebook: devops_grpo_training.ipynb
• —OpenEnv manifest: openenv.yaml

TL;DR

This environment exists to teach a capability that current LLMs still struggle with:

stateful operational reasoning under uncertainty

The goal is not to build an unsafe “AI that runs production alone.” The goal is to train and evaluate incident copilots that can:

• —investigate before acting
• —reason about hidden causes from partial evidence
• —avoid unsafe remediations
• —maintain causal context across multiple steps
• —verify customer recovery instead of stopping at plausible advice

Why This Problem Matters

Existing DevOps copilots mostly answer questions, retrieve docs, and suggest likely root causes.

Real on-call work is harder:

• —the true cause is only partially visible
• —many actions look plausible
• —the wrong action order can worsen the outage
• —mitigation changes the world and must be re-evaluated
• —success is not “sounding smart,” but restoring service safely

That gap is exactly what this environment targets.

This makes the benchmark useful for:

• —SRE copilots
• —platform automation safety research
• —offline evaluation of operational agents
• —training junior on-call engineers in a safe sandbox
• —long-horizon world-modeling research for LLMs

What Makes This Submission Ambitious

The hackathon explicitly rewards environments that are ambitious, original, and genuinely useful for training.

This environment is ambitious in the right way:

1. 1.It is interactive, not static. The agent is evaluated over trajectories, not single-turn answers.

1. 1.It is partially observable. The model must infer hidden causes from logs, metrics, dependency signals, and incident history.

1. 1.It is consequence-driven. Actions change the world. Wrong mitigations waste steps, create risk, and reduce final score.

1. 1.It blends multiple real capabilities. The benchmark requires diagnosis, sequencing, safety, recovery confirmation, and lightweight operational communication.

1. 1.It is hard to game. Fluent but shallow outputs do not score well unless they lead to correct behavior.

Environment Overview

The simulated production world contains six services:

The environment is seeded and reproducible. Each task starts from a realistic incident brief plus observable service state.

The agent sees:

• —service health snapshots
• —error, latency, and saturation signals
• —recent events
• —discovered facts
• —available commands
• —task-specific operator hints

The agent acts through operational commands such as:

• —query_logs
• —get_metrics
• —get_recent_deploys
• —inspect_config
• —check_dependency
• —rollback_deploy
• —restart_service
• —scale_service
• —clear_connections
• —disable_feature_flag
• —drain_queue
• —failover_db
• —run_smoke_test
• —post_status_update

Task Suite

The benchmark includes six incident families:

For training stability, the main Colab notebook also supports a 3-task showcase subset:

• —bad_auth_deploy
• —db_pool_exhaustion
• —network_partition_failover

This gives a cleaner before/after training story while preserving the full environment for broader evaluation.

Why OpenEnv Is The Right Fit

This problem cannot be faithfully captured by a static dataset.

A useful benchmark here needs:

• —seeded resets
• —stepwise world transitions
• —typed observations, actions, rewards, and graders
• —reproducible scoring
• —trajectory-level evaluation

That is exactly what OpenEnv provides.

Exposed API surface:

• —POST /reset?task_id=<id>&seed=<int>
• —POST /step
• —GET /state
• —GET /grader
• —GET /tasks
• —POST /baseline

Implementation entrypoints:

• —API server: server/app.py
• —environment logic: server/environment.py
• —world and scenario metadata: server/scenarios.py
• —reward and grading logic: server/rewards.py
• —typed models: server/models.py

Reward Design

The reward is designed to teach behavior that matters in real incidents:

• —reward for meaningful investigation
• —higher reward for correct mitigations
• —strong reward for verified recovery
• —penalties for unsafe, repeated, or irrelevant actions
• —penalties for wasting steps

The grader also tracks final behavior quality through:

• —score
• —resolved
• —root_cause_fixed
• —recovery_confirmed
• —unsafe_actions
• —ineffective_actions
• —investigation_coverage
• —communication_sent

The key principle is simple:

The model should not get high score unless it behaves like a good incident responder.

Training And Evaluation Story

This repository includes Colab-friendly training notebooks built with Unsloth and HF TRL, as required by the hackathon.

1. Main Training Notebook

devops_grpo_training.ipynb

This notebook:

• —connects directly to the hosted Hugging Face Space
• —builds a teacher-rolled stepwise dataset from environment trajectories
• —evaluates the zero-shot base model
• —performs a strong SFT warm start
• —optionally refines with a short GRPO pass
• —saves evaluation JSON outputs

2. Blog

Blog.MD

This is the short narrative writeup for judges and readers who want the motivation, environment story, and problem framing without reading the full code.

What We Measure

We do not rely on a single reward curve alone.

The environment is designed to support meaningful behavioral metrics such as:

• —average validation score
• —solve rate
• —investigation coverage
• —unsafe actions per episode
• —action validity and ordering quality

The goal is to show that the trained policy improves on the dimensions that matter operationally, not just on superficial text quality.

Current Reference Point

The environment is solvable and reproducible.

The in-notebook heuristic reference policy reaches:

• —average validation score: 0.896
• —solve rate: 1.0

This is useful as an operational ceiling and as a sanity check that the tasks are not impossible.

Why This Is Useful Beyond The Hackathon

This is not just a demo or an API wrapper.

It is a reusable environment for studying one of the most important applied AI questions:

Can language agents make better decisions in high-stakes, partially observable professional workflows?

Possible downstream uses:

• —benchmark for incident-response copilots
• —offline evaluation for operational agents before deployment
• —research on long-horizon causal reasoning
• —safe simulation environment for automation policy learning

Running The Environment

Local Python

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip install -e .
uvicorn server.app:app --host 0.0.0.0 --port 7860

Docker

docker build -t devops-incident-responder .
docker run --rm -p 7860:7860 devops-incident-responder

Quick API Smoke Test

curl -X POST "http://localhost:7860/reset?task_id=bad_auth_deploy&seed=101"
curl -X POST "http://localhost:7860/step" \
  -H "Content-Type: application/json" \
  -d '{"command":"get_recent_deploys","target":"system"}'
curl "http://localhost:7860/grader"

Validation

Recommended pre-submission flow:

docker build -t devops-incident-responder .
python3 inference.py
openenv validate
./testing/val.sh https://tek-wizard-devops-incident-responder.hf.space .

Project Structure

.
├── Blog.MD
├── Dockerfile
├── inference.py
├── openenv.yaml
├── README.md
├── devops_grpo_training.ipynb
├── server/
│   ├── app.py
│   ├── environment.py
│   ├── models.py
│   ├── rewards.py
│   ├── scenarios.py
│   ├── seed_sets.py
│   ├── session_store.py
│   └── tasks.py
└── testing/
    └── val.sh

Closing

We believe the next generation of useful AI systems will not win by being more verbose.

They will win by being:

• —safer
• —more stateful
• —more causally grounded
• —better at acting under uncertainty

DevOps Incident Responder is our attempt to build exactly the kind of environment needed to train that behavior.

Because in a real incident, the goal is not to sound helpful.

The goal is to restore service without making the outage worse.