Apppublic

Ayushmaan03/log-analysis-agent

sourceHugging Faceupdated 6mo agoView on Hugging Face

2likes

App README

Log Analysis & Incident Diagnosis Agent (OpenEnv)

Description

This OpenEnv environment simulates a real-world Site Reliability Engineering (SRE) task. An agent is presented with a stream of server logs and telemetry metrics (CPU, memory, latency). The agent must use a progressive log reveal mechanism to read through the logs, detect anomalies, classify the root cause, and deploy the correct mitigation strategy.

Motivation

Incident response and log analysis are critical tasks for modern SREs. This environment provides a safe, deterministic, and reproducible sandbox for evaluating LLM agents on their reasoning, debugging, and log-interpretation capabilities.

Action Space

The agent can take 4 discrete categorical actions: | Action | Description | |---|---| | request_more_logs() | Reveals the next 2 lines of logs (mimics pagination/log scrolling). | | detect('anomaly'\|'normal') | Predicts whether the system has an anomaly or is operating normally. | | classify('<issue_type>') | Classifies the root cause. Valid values: cpu_spike, memory_leak, disk_full, network_latency, service_crash. | | mitigate('<action>') | Attempts a fix. Valid values: restart_service, scale_up, clear_cache, rollback, no_action. |

Observation Space

Field	Type	Description
`logs`	`List[str]`	Currently visible log lines (progressively revealed).
`metrics`	`Dict[str, float]`	CPU usage (%), Memory usage (%), Latency (ms).
`timestep`	`int`	Current step count within the episode.
`max_steps`	`int`	Maximum allowed steps (10).
`info`	`str`	Text feedback from the previous action.

Tasks & Difficulties

#	Task	Difficulty	Grader
1	Anomaly Detection	🟢 Easy	`1.0` if `detect()` matches ground truth, else `0.0`.
2	Root Cause Classification	🟡 Medium	`1.0` if `classify()` matches ground truth issue type (or correct `detect('normal')` for normal scenarios), else `0.0`.
3	Full Incident Resolution	🔴 Hard	Weighted composite: `0.7 × (0.3×detect + 0.4×classify + 0.3×mitigate) + 0.3×efficiency`.

Reward Function

Action	Condition	Reward
`request_more_logs`	Any	-0.1 (time penalty)
`detect`	Correct	+1.0
`detect`	Incorrect	-1.0
`classify`	Correct	+2.0
`classify`	Incorrect	-1.0
`mitigate`	Correct	+2.0
`mitigate`	Incorrect/harmful	-2.0 (episode ends)
Timeout	Max steps reached	-2.0

Setup Instructions

bash

# 1. Install dependencies
pip install -r requirements.txt

# 2. Start the FastAPI server
uvicorn server.app:app --port 8000

# 3. Validate (in a separate terminal, from the project root)
openenv validate

Baseline Evaluation

Environment variables required (copy .env.example .env and fill in):

—API_BASE_URL — LLM API endpoint (default: Groq)
—MODEL_NAME — Model identifier (default: llama-3.3-70b-versatile)
—HF_TOKEN or GROQ_API_KEY — Your API key

bash

python inference.py

Baseline Scores (llama-3.3-70b-versatile, 10 seeded episodes, seeds 42–51)

Task	Score
Task 1 – Anomaly Detection (easy)	0.90
Task 2 – Root Cause Classification (medium)	0.70
Task 3 – Full Incident Resolution (hard)	0.52

Scores are reproducible by running python inference.py with the same model and seeds.

Docker

bash

docker build -t log-analysis-agent .
docker run -p 8000:8000 log-analysis-agent