Ginnipahwa05/Meta-Pytorch

Distributed Incident War Room Environment

A production-ready OpenEnv environment for real-world SRE debugging simulation. An AI agent operates as a Site Reliability Engineer inside a live war room, actively operating distributed microservices systems during simultaneous production incidents.

Quick Start

Choose the path that matches how you want to use the project:

• —Run the dashboard locally

  python3 -m venv .venv
  source .venv/bin/activate
  pip install -r requirements.txt
  cp .env.example .env
  PYTHONPYCACHEPREFIX=/tmp/pycache python -m uvicorn app:app --host 0.0.0.0 --port 7860

Then open http://localhost:7860.

• —Run the one-command local workflow

  ./run-local.sh

This starts the FastAPI app and uses one of three log modes:

• —local Elasticsearch if available
• —remote Elasticsearch if ELASTICSEARCH_URL is set
• —local fallback/demo logs if no external log backend is configured

• —Deploy to Hugging Face Spaces
• —Use the included Dockerfile
• —Keep the README front matter at the top of this file
• —Point the Space at a repo containing app.py, Dockerfile, requirements.txt, and scripts/seed_project_errors_to_elastic.py
• —See DEPLOYMENT.md for the hosted setup checklist

Overview

Problem Statement

Modern distributed microservices systems can fail in multiple places at once. Engineers must:

• —Quickly identify root causes with partial information
• —Navigate misleading signals and noisy logs
• —Make critical decisions under time pressure
• —Balance tradeoffs between fixing one issue and preventing others

Our Solution

A realistic simulator where an AI agent:

1. 1.Observes alerts, metrics, and logs (partial and noisy)
2. 2.Investigates using structured debugging actions
3. 3.Diagnoses root causes systematically
4. 4.Resolves incidents with correct sequence of fixes
5. 5.Minimizes system damage and downtime

Real-World Inspiration

This environment is inspired by actual production incidents like:

• —Database connection leaks → cascading latency across services
• —Bad deployments → sudden error rate spikes
• —Cache failures → downstream service overload
• —Traffic spikes → insufficient scaling → timeouts
• —Replication lag → data inconsistency cascades

Architecture

Microservices

┌─────────────────────────────────────────┐
│          API Gateway (Entry Point)       │
├─────────────────────────────────────────┤
│ Auth Service │ Payments │ Cache Backend │
├─────────────────────────────────────────┤
│        Database (Primary/Replica)        │
└─────────────────────────────────────────┘

System Properties

• —Multiple simultaneous incidents (especially hard difficulty)
• —Partial observability (logs only visible after querying)
• —Noisy signals (misleading logs mixed with real alerts)
• —Cascading failures (fixing one service may affect others)
• —Time pressure (system degrades each step if unresolved)
• —Tradeoffs (quick fix vs proper solution)

Action Space

Agents can take 8 structured actions:

Observation Space

Observable state includes:

• —Active alerts (severity, service, message)
• —Service status (latency, error rate, CPU, memory)
• —Recent logs (partial view, ~20 entries)
• —Active incident IDs
• —Metrics summary
• —Current step & damage score

Hidden state includes:

• —True root causes
• —Complete log history
• —Dependency graph
• —System damage timeline

Reward Function

Continuous rewards in range [-1.0, 1.0]:

Positive Rewards

• —+0.1: Useful exploration (querying logs/metrics)
• —+0.15: Finding anomalous metrics
• —+0.25: Symptom fix (restarting service)
• —+0.5: Root cause fix (appropriate rollback)
• —+0.6: Correct diagnosis
• —+0.2: Strategic scaling

Negative Rewards

• —-0.1: Unknown actions
• —-0.2: Unnecessary service restart
• —-0.3: Incorrect diagnosis
• —-0.15: Ineffective rollbacks

Damage Penalties

• —System degrades 0.02 per step per unresolved incident
• —Restarting healthy services increases damage
• —Damage is cumulative

Tasks

Easy (1 Incident, Clear Signals)

• —Database Connection Leak: Single clear incident, straightforward logs
• —Bad Deployment: Recent bad code deploy, obvious error logs
• —Goal: Identify the failing service in <15 steps
• —Sample Score: 0.85+

Medium (2 Incidents, Misleading Signals)

• —Cache Failure + API Overload: Cache backend down triggers cascade
• —Traffic Spike + Insufficient Scaling: Load increases beyond capacity
• —Goal: Navigate misleading logs, identify correct root cause in <25 steps
• —Sample Score: 0.75+

Hard (3 Incidents, Complex Tradeoffs)

• —Cascading Auth + Payment Failures: Auth bug cascades to payments
• —Multi-Service Outage: DB replication lag requires careful resolution choice
• —Goal: Resolve all incidents optimally, handle tradeoffs in <30 steps
• —Sample Score: 0.65+

Grading

Deterministic scoring with 3 components:

Final Score = 0.5 × Correctness + 0.3 × Efficiency + 0.2 × Damage

Example:

• —Correct diagnosis: correctness = 0.9
• —20 steps of 30: efficiency = (1 - 20/30) = 0.33
• —Damage 0.15: damage = (1 - 0.15) = 0.85
• —Score = 0.5(0.9) + 0.3(0.33) + 0.2(0.85) = 0.719

Grader Variants

Each difficulty level has appropriate grading:

• —Easy: Focus on correctness + basic efficiency
• —Medium: Penalize incorrect diagnoses, reward systematic exploration
• —Hard: Heavy penalty for inefficiency, bonus for optimal sequence

Repository Layout

Meta PyTorch/
├── app.py                               # FastAPI app and dashboard UI
├── environment.py                       # Core simulator implementation
├── tasks.py                             # Incident scenarios and task catalog
├── graders.py                           # Deterministic grading logic
├── inference.py                         # Baseline inference runner
├── models.py                            # Pydantic request/response models
├── scripts/seed_project_errors_to_elastic.py
│                                       # Demo log seeding utility
├── reference-stack/                     # Optional sidecar demo services
├── openenv.yaml                         # OpenEnv specification
├── Dockerfile                           # Hugging Face / container deployment
├── run-local.sh                         # Local bootstrap helper
├── stop-local.sh                        # Local shutdown helper
├── requirements.txt                     # Python dependencies
└── README.md                            # Project documentation

Installation

Local Setup

# Clone the repository
git clone https://github.com/garimapahwa/Meta-PyTorch.git
cd Meta-PyTorch

# Create a virtual environment
python3 -m venv .venv
source .venv/bin/activate  # on Windows: .venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Create local config
cp .env.example .env

# Start the app
PYTHONPYCACHEPREFIX=/tmp/pycache python -m uvicorn app:app --host 0.0.0.0 --port 7860

Open http://localhost:7860 after the server starts.

Minimal .env setup

The app works without external observability credentials by falling back to simulator data and local replay logs. Start with .env.example, then optionally add:

DD_API_KEY=""
DD_APP_KEY=""
DD_SITE="datadoghq.com"

ELASTICSEARCH_URL=""
ELASTICSEARCH_API_KEY=""
ELASTICSEARCH_USERNAME=""
ELASTICSEARCH_PASSWORD=""
ELASTICSEARCH_LOG_INDEX="logs-*"
ELASTICSEARCH_TIMESTAMP_FIELD="@timestamp"
ELASTICSEARCH_SERVICE_FIELD="service"
ELASTICSEARCH_MESSAGE_FIELD="message"
ELASTICSEARCH_LEVEL_FIELD="log.level"
ELASTICSEARCH_VERIFY_TLS="true"

You only need Datadog or Elasticsearch variables if you want live external logs, metrics, or traces.

Docker Setup (HF Space Compatible)

# Build image
docker build -t devops-war-room:latest .

# Run server
docker run -p 7860:7860 \
  -e PORT=7860 \
  devops-war-room:latest

# Test
curl http://localhost:7860/ping

Usage

Glass Dashboard

Open http://localhost:7860/ to launch the glassmorphism war-room UI. The dashboard lets you:

• —Reset and step the simulated incident environment
• —Inspect current state and grades
• —Pull logs from Elasticsearch or Datadog
• —Pull metrics and APM traces from Datadog
• —Fall back to local simulator signals if external credentials are not configured

Recommended Local Workflows

• —Fastest path for app-only development

  source .venv/bin/activate
  PYTHONPYCACHEPREFIX=/tmp/pycache python -m uvicorn app:app --host 0.0.0.0 --port 7860 --reload

• —One-command bootstrap

  ./run-local.sh

This is the best option when you want the app plus optional Elasticsearch integration with the least setup friction.

• —Local fallback mode If you do not configure Datadog or Elasticsearch, the dashboard still works using:
• —simulator logs from the environment
• —locally replayed incident logs from .run/local-demo-logs.jsonl
• —synthetic traces derived from the available local signals

Common API Flow

If you are using the simulator programmatically or from curl, this is the usual sequence:

1. 1.Reset the environment with a task ID.
2. 2.Inspect logs, metrics, traces, or state.
3. 3.Step through actions until the incident is resolved.
4. 4.Read the final grade.

Example:

curl -X POST http://localhost:7860/reset \
  -H "Content-Type: application/json" \
  -d '{"task_id": "easy_0"}'

curl http://localhost:7860/state

curl -X POST http://localhost:7860/step \
  -H "Content-Type: application/json" \
  -d '{"action_type": "QUERY_LOGS", "service": "db"}'

curl http://localhost:7860/grade

Datadog Access

Set these environment variables to connect the dashboard to Datadog:

• —DD_API_KEY
• —DD_APP_KEY
• —DD_SITE such as datadoghq.com or datadoghq.eu
• —DD_LOG_INDEXES optionally narrows the log search to a comma-separated set of indexes

Datadog-backed API routes:

• —/api/logs
• —/api/metrics
• —/api/apm

If credentials are missing, these endpoints automatically return local simulator data so the UI continues to function. The app now loads .env automatically on startup, so local Datadog credentials work without manual export commands.

Elasticsearch Access

Set these environment variables to use Elasticsearch for dashboard logs:

• —ELASTICSEARCH_URL
• —ELASTICSEARCH_API_KEY or ELASTICSEARCH_USERNAME and ELASTICSEARCH_PASSWORD
• —ELASTICSEARCH_LOG_INDEX such as logs-*
• —ELASTICSEARCH_TIMESTAMP_FIELD if your time field is not @timestamp
• —ELASTICSEARCH_SERVICE_FIELD if your service field is not service
• —ELASTICSEARCH_MESSAGE_FIELD if your message field is not message
• —ELASTICSEARCH_LEVEL_FIELD if your level field is not log.level

Observability routes:

• —/api/observability/status
• —/api/logs
• —/api/metrics
• —/api/apm

When ELASTICSEARCH_URL is configured, /api/logs uses Elasticsearch first. Metrics and APM remain Datadog-backed unless you extend those routes too.

Choosing a Log Backend

The app resolves providers in this order:

1. 1.Elasticsearch for /api/logs when ELASTICSEARCH_URL is configured
2. 2.Datadog for logs, metrics, and traces when Datadog credentials are configured
3. 3.Local fallback data when no external backend is available

This means the dashboard stays usable in all three situations:

• —fully local demo mode
• —partially connected mode with Datadog only
• —hosted mode with a reachable Elasticsearch cluster

Running Inference Script

python inference.py

Output format (required):

[START]
{"task": "easy_0", "difficulty": "easy", ...}
[STEP]
{"step": 1, "action": "query_logs", "reward": 0.1, ...}
[STEP]
{"step": 2, "action": "resolve_incident", "reward": 0.6, ...}
[END]
{"status": "completed", "steps_taken": 2, "final_score": 0.85, ...}

Using as Library

from environment import make_env
from models import Action, ActionType, ServiceName

# Create environment
env = make_env(task_id="easy_0", seed=0)

# Reset
obs = env.reset()

# Take action
action = Action(
    action_type=ActionType.QUERY_LOGS,
    service=ServiceName.DB,
)
obs, reward, done, info = env.step(action)

# Get grade
grade = env.get_grade()
print(f"Score: {grade['score']:.4f}")

Using FastAPI Server

# Start server
python -m uvicorn app:app --host 0.0.0.0 --port 7860

# Health check
curl http://localhost:7860/ping

# Reset environment
curl -X POST http://localhost:7860/reset \
  -H "Content-Type: application/json" \
  -d '{"task_id": "easy_0"}'

# Step
curl -X POST http://localhost:7860/step \
  -H "Content-Type: application/json" \
  -d '{
    "action_type": "QUERY_LOGS",
    "service": "db"
  }'

# Get state
curl http://localhost:7860/state

# Get grade
curl http://localhost:7860/grade

# List tasks
curl http://localhost:7860/tasks

One-Command Local Start

If you want the local Elasticsearch node and the FastAPI app to come up together:

./run-local.sh

This script:

• —starts Elasticsearch on 127.0.0.1:9200 if it is not already running and the local distribution is available
• —starts the app on 127.0.0.1:7860 if it is not already running
• —uses ELASTICSEARCH_URL from .env when you point at a remote cluster
• —falls back to local replayed/demo logs when no Elasticsearch backend is available
• —waits for the app, and for Elasticsearch only when it is expected to start locally

To stop the processes started by the script:

./stop-local.sh

Set SHUTDOWN_TIMEOUT=15 if you want the stop script to wait longer before force-killing a stuck process.

Notes:

• —if elasticsearch-9.3.2/ is present locally, the script can start a local Elasticsearch node
• —if ELASTICSEARCH_URL points to a remote cluster, the script skips local Elasticsearch startup
• —if neither is available, the script still starts the app in fallback mode
• —the app log is written to .run/app.log

Reference Harness

If you want a small broken service that emits real JSON logs into Elasticsearch for the dashboard to inspect:

docker compose -f reference-stack/docker-compose.yml up --build

That stack brings up:

• —the dashboard on http://localhost:7860
• —a deliberately flaky orders-service on http://localhost:8081
• —Elasticsearch on http://localhost:9200

The dashboard is preconfigured to read the harness logs from broken-ref-logs-*. For usage details and sample queries, see:

reference-stack/README.md

Project Error Replay

If you want to replay this repo's own failure modes into Elasticsearch and diagnose them in the dashboard:

.venv/bin/python scripts/seed_project_errors_to_elastic.py --scenario all

If ELASTICSEARCH_URL is configured, this seeds Elasticsearch directly. If not, it writes a local replay file under .run/local-demo-logs.jsonl, and the dashboard will surface those incidents through the existing /api/logs fallback.

This seeds structured demo incidents such as:

• —missing local Elasticsearch install for run-local.sh
• —port 7860 already in use
• —missing Docker for the reference-stack flow

The seeded logs use:

• —service: meta-pytorch-demo
• —incident IDs: INC-DEMO-*

Suggested dashboard filters:

• —query: incident_id:INC-DEMO-*
• —service: meta-pytorch-demo

For a side-by-side terminal layout, run:

./demo-side-by-side.sh

Hosting

For a public deployment path and production checklist, see:

DEPLOYMENT.md

Hugging Face Spaces Notes

For the Space to boot correctly, the repo used by Hugging Face should contain:

• —README.md with the Space front matter block at the top
• —Dockerfile
• —app.py
• —requirements.txt
• —scripts/__init__.py
• —scripts/seed_project_errors_to_elastic.py

Use the Hugging Face Space URL for demos:

https://huggingface.co/spaces/Ginnipahwa05/Meta-Pytorch

Endpoints

Health & Status

• —GET / - Service info
• —GET /ping - Health check (HTTP 200)
• —GET /health - Detailed health
• —GET /api/observability/status - Active provider status and connectivity hints

Core OpenEnv

• —POST /reset - Initialize environment, return observation
• —POST /step - Execute action, return (obs, reward, done, info)
• —GET /state - Get current partially observable state

Metadata

• —GET /tasks - List all tasks
• —GET /tasks/{task_id} - Get task details
• —GET /grade - Get final grade (after episode done)

Observability & Demo Data

• —GET /api/logs - Query Elasticsearch, Datadog, or local fallback logs
• —GET /api/metrics - Query Datadog or simulator-backed metrics
• —GET /api/apm - Query Datadog traces, Elastic-derived traces, or fallback traces
• —POST /api/demo/seed-logs - Seed demo incident logs into Elasticsearch when configured

Validation Checklist

• —[x] HF Space Compatible: Dockerfile works, server on port 7860
• —[x] OpenEnv Spec: step(), reset(), state() implemented
• —[x] HTTP 200: /ping endpoint returns 200
• —[x] Docker: docker build and docker run succeed
• —[x] Inference: inference.py runs, produces deterministic scores
• —[x] Logging: Exact format [START], [STEP], [END]
• —[x] Tasks: 3+ tasks (easy, medium, hard)
• —[x] Graders: Deterministic, emitted scores stay strictly between 0 and 1
• —[x] Typed Models: Pydantic Action, Observation, Reward
• —[x] OpenAI Client: Used for LLM calls
• —[x] Env Vars: APIBASEURL, MODELNAME, HFTOKEN supported
• —[x] Runtime: < 20 minutes on 2vCPU/8GB

Baseline Scores

Reference Results (seed=0):

• —easy_0: ~0.85 (correct diagnosis fast)
• —easy_1: ~0.82
• —medium_0: ~0.68 (navigates misleading logs)
• —medium_1: ~0.65
• —hard_0: ~0.52 (complex tradeoffs)
• —hard_1: ~0.48

Agents achieving >0.70 on hard difficulty are performing exceptionally.

Reproducibility

• —Deterministic: Same seed produces same incidents, logs, metrics
• —Seed usage: make_env(task_id="easy_0", seed=42)
• —No randomness in grading: Scores deterministic given episode
• —Fixed scenario sets: Easy/Medium/Hard have predefined scenarios

Performance Characteristics

• —Environment creation: <100ms
• —Step execution: <50ms average
• —Full episode (30 steps): <2 seconds
• —Full inference run (all 6 tasks): <20 seconds

Competition Features

• —Realistic: Inspired by real SRE debugging workflows
• —Complex: Multiple simultaneous incidents with cascading failures
• —Challenging: Misleading signals, partial observability, time pressure
• —Fair: Deterministic grading, multiple difficulty levels
• —Extensible: Easy to add scenarios, modify incidents, adjust rewards

Future Extensions

• —Network delay simulation
• —Resource contention modeling
• —More sophisticated failure modes
• —Multi-agent coordination scenarios
• —Continuous reward shaping

License

This project is open source for hackathon evaluation.

Built for Maximum Impact 🚀

• —✅ Passes all validation
• —✅ Production-ready code
• —✅ Real-world relevance
• —✅ Top-tier hackathon submission