Apppublic

DevilNReality/Incident-Triage-Reinforcement-Learning

sourceHugging Faceupdated 6mo agoView on Hugging Face

0likes

App README

🚨 Incident Triage & Alert Routing — OpenEnv

Meta × Hugging Face AI Hackathon | Round 1 An OpenEnv environment for training AI agents to triage production incidents like a senior SRE.

Environment Description

This environment simulates the real-world job of a Site Reliability Engineer (SRE) who receives a flood of production alerts and must:

1.Identify the root cause from a set of correlated alerts
2.Assign the correct severity level (P1–P4)
3.Route the incident to the correct on-call team (SRE, Database, Network, or App)
4.(Hard task only) Suppress noise alerts — flapping/self-resolving false positives

Every tech company faces alert fatigue. Reducing Mean Time To Detection (MTTD) by even 5 minutes has enormous operational value. This environment benchmarks how well an AI agent handles that pressure.

Observation Space

At each step the agent receives an AlertObservation:

python

class AlertObservation(BaseModel):
    alerts:       List[Alert]        # incoming monitoring alerts
    services_map: Dict[str, str]     # service → responsible team
    step:         int                # current step in the episode
    max_steps:    int                # maximum steps allowed
    task_id:      str                # "task_easy" | "task_medium" | "task_hard"

Each Alert contains:

python

class Alert(BaseModel):
    alert_id:   str    # e.g. "alert-001"
    service:    str    # e.g. "postgres-primary"
    metric:     str    # e.g. "db_query_time_ms"
    value:      float  # current metric value
    threshold:  float  # value that triggered the alert
    duration_s: int    # how long it has been firing (seconds)

Services Map

Service	Team
`api-server-03`, `api-server-07`, `redis-cache-01`	`sre`
`postgres-primary`, `postgres-replica`	`database`
`checkout-service`, `payment-service`, `auth-service`, `reporting-service`	`app`
`load-balancer-01`, `cdn-edge-02`, `dns-resolver-01`	`network`

Action Space

python

class TriageAction(BaseModel):
    severity:   Literal['P1', 'P2', 'P3', 'P4']
    team:       Literal['sre', 'database', 'network', 'app']
    root_cause: str        # one-sentence explanation, graded by keyword match
    suppressed: List[str]  # alert_ids believed to be noise (task_hard only)

Field	Description
`severity`	P1 = Critical (revenue impact), P2 = High (users affected), P3 = Medium, P4 = Low
`team`	On-call team to page: `sre`, `database`, `network`, or `app`
`root_cause`	Free-text root cause. Scored by keyword match against ground truth.
`suppressed`	Alert IDs the agent believes are noise. Only scored in `task_hard`.

Reward Function

task_easy

Component	Weight
Severity correct	0.50
Team correct	0.50

task_medium

Component	Weight
Severity correct	0.35
Team correct	0.35
Root cause keywords	0.30

task_hard

Component	Weight
Noise suppression (F1)	0.20
Severity correct	0.25
Team correct	0.25
Root cause keywords	0.30
Over-suppression penalty	−0.20 per real alert wrongly suppressed

Task Descriptions

🟢 task_easy — Single Alert, Obvious Root Cause

—1 alert randomly chosen from 4 scenario variants
—Variants: CPU spike → SRE · DB timeout → Database · Network latency → Network · 5xx errors → App
—Max steps: 3
—Expected score (frontier model): 0.9–1.0

🟠 task_medium — 3 Correlated Alerts, Identify Root Cause

—3 correlated alerts from 4 scenario sets (Database overload, Network partition, App memory leak, CPU saturation)
—Agent must identify the single upstream root cause from downstream effects
—Max steps: 3
—Expected score (frontier model): 0.5–0.75

🔥 task_hard — 10–15 Alerts, Suppress Noise & Escalate

—6 real alerts (database outage root cause) + 5 noise alerts (short-duration, barely over threshold)
—Noise rule: duration_s < 60 AND value < threshold × 1.10
—Agent must suppress noise alerts and correctly identify the P1 database root cause
—Max steps: 5
—Expected score (frontier model): 0.2–0.45

API Reference

All endpoints are available on port 7860 (via nginx reverse proxy).

Method	Path	Description
`GET`	`/health`	Health check — returns `{"status": "ok"}`
`GET`	`/tasks`	List all tasks with metadata
`POST`	`/reset`	Start a new episode. Body: `{"task_id": "task_easy", "seed": null}`
`POST`	`/step`	Submit a `TriageAction`. Returns observation, reward, done, info.
`GET`	`/state`	Full environment state including ground truth team/severity

Example: Full Episode

python

import requests

BASE = "https://<your-username>-incident-triage-env.hf.space"

# 1. Reset
obs = requests.post(f"{BASE}/reset", json={"task_id": "task_medium"}).json()

# 2. Step
action = {
    "severity": "P1",
    "team": "database",
    "root_cause": "Database overload causing query timeouts and downstream 5xx errors",
    "suppressed": []
}
result = requests.post(f"{BASE}/step", json=action).json()
print(result["reward"], result["done"], result["info"])

🚀 Setup & Deployment

Option A — Hugging Face Spaces (recommended)

Prerequisites

—Git installed
—A Hugging Face account with a token from https://huggingface.co/settings/tokens

Step 1 — Install the HF CLI & login

bash

pip install huggingface_hub
huggingface-cli login

Step 2 — Create a new Space

Go to https://huggingface.co/new-space and set:

—Space name: incident-triage-env
—SDK: Docker
—Visibility: Public

Step 3 — Clone and push

bash

git clone https://huggingface.co/spaces/<your-username>/incident-triage-env
cd incident-triage-env

# Copy all project files here, then:
git add .
git commit -m "deploy: Incident Triage OpenEnv"
git push

HF Spaces detects the Dockerfile and builds automatically (~3–5 min). Monitor progress in the Logs tab.

Step 4 — Add secrets (for LLM agent mode)

In your Space → Settings → Repository secrets:

Secret	Value
`HF_TOKEN`	Your HF token (`hf_...`)
`API_BASE_URL`	`https://router.huggingface.co/v1`
`MODEL_NAME`	`meta-llama/Llama-3.3-70B-Instruct`

The Streamlit UI and rule-based agent work without secrets. Secrets are only needed for LLM Agent mode.

Option B — Local Development

bash

# 1. Install dependencies
pip install -r requirements.txt

# 2. Start FastAPI backend (terminal 1)
cd backend
uvicorn server:app --reload --port 8000

# 3. Start Streamlit UI (terminal 2, from project root)
BACKEND_URL=http://localhost:8000 streamlit run frontend/app.py
# Opens at http://localhost:8501

# 4. (Optional) Run the LLM baseline agent
cp .env.example .env   # fill in your API keys
source .env
python inference.py

Option C — Docker locally

bash

docker build -t incident-triage-env .
docker run -p 7860:7860 \
  -e HF_TOKEN=hf_... \
  -e API_BASE_URL=https://router.huggingface.co/v1 \
  -e MODEL_NAME=meta-llama/Llama-3.3-70B-Instruct \
  incident-triage-env

# UI + API both at http://localhost:7860

Architecture

Browser / OpenEnv Checker
         │
         ▼ port 7860 (public)
       nginx
      ╱     ╲
     ▼         ▼
FastAPI       Streamlit
 :8000         :8501
(API calls)   (Web UI)

nginx routes:

—POST /reset, GET /health, /tasks, /step, /state → FastAPI (OpenEnv checker)
—/ and all other traffic → Streamlit UI (browser)

Project Structure

incident-triage-env/
├── Dockerfile           ← Builds & runs nginx + FastAPI + Streamlit
├── nginx.conf           ← Reverse proxy: port 7860 → FastAPI:8000 / Streamlit:8501
├── start.sh             ← Startup script launching all three processes
├── requirements.txt     ← Python dependencies (latest versions)
├── openenv.yaml         ← OpenEnv spec: tasks, action/observation spaces, reward
├── pyproject.toml       ← Package config; server entry point: server.app:main
├── .env.example         ← Template for local env vars
├── .dockerignore        ← Excludes __pycache__, *.pyc, .git, uv.lock etc.
├── inference.py         ← LLM baseline agent (uses OpenAI client + HF router)
├── backend/
│   ├── models.py        ← Pydantic v2: Alert, AlertObservation, TriageAction, TriageReward
│   ├── tasks.py         ← All task definitions + graders (easy / medium / hard)
│   ├── environment.py   ← OpenEnv contract: reset() step() state()
│   └── server.py        ← FastAPI app with /reset /step /state /health /tasks
├── server/
│   └── app.py           ← Entry point wrapper for multi-mode deployment (server.app:main)
└── frontend/
    └── app.py           ← Streamlit UI: alert cards, routing table, score chart

Baseline Scores

Task	Score	Notes
`task_easy`	~0.95	Near-perfect on single alert
`task_medium`	~0.63	Good severity+team, partial root cause keyword match
`task_hard`	~0.31	Noise suppression is the primary challenge

Scores from meta-llama/Llama-3.3-70B-Instruct via HF inference router.

Built for the Meta × Hugging Face AI Hackathon, Round 1.