channu07/microgrid-env-v2

MicrogridEnv v2

A stochastic reinforcement learning benchmark for optimizing distributed energy systems under uncertainty.

Designed for RL research, benchmarking, and real-world energy optimization simulations. This environment is a benchmark, not just a simulation.

TL;DR (30-second read)

• —What: RL environment where an agent operates a microgrid under noisy solar, wind, load, and equipment faults
• —State: 12 dimensions (time, load, solar, wind, battery, price, frequency, voltage, fault, 3 component healths)
• —Action: 4 dimensions (battery dispatch, grid import, load shed, dispatch priority)
• —Reward: multi-objective — survival + stability − cost − outage − wear
• —Tasks: 3 tasks with clear difficulty progression (easy → medium → hard)
• —APIs: Gymnasium (gym.Env), REST, WebSocket, MCP
• —Stochastic environment, deterministic graders — stochastic dynamics for training, seedable graders for reproducible benchmarking
• —Baselines included: Random, Greedy, DoNothing, SmartHeuristic

System Overview

 Solar (stochastic) ──┐
 Wind  (stochastic) ──┤
                      ├──► [ Microgrid Agent ] ──► Actions
 Load  (stochastic) ──┤         (you)              ├── battery_power
 Grid  (TOU price)  ──┘                            ├── grid_import
                                                   ├── load_shed
 Faults (random)    ──────────────────────────────►└── dispatch_priority

 Outputs: frequency, voltage, cost, unserved MWh, component wear

What Makes MicrogridEnv v2 Unique

• —Stochastic dynamics + deterministic grading — rare combination: train under realistic noise, benchmark under reproducible seeds
• —Real power-system physics — frequency droop, voltage droop, battery efficiency, component degradation
• —Multi-task benchmark, not a single scenario — three distinct task types stress different skills
• —Component wear modeled — agents that burn equipment lose over long horizons; few RL envs capture this
• —Hard terminal conditions — frequency excursion > 1.5 Hz triggers blackout, creating real survival pressure

Problem Definition

Real microgrid operators continuously balance generation (solar, wind, battery) against demand under three sources of uncertainty: weather-driven renewable intermittency, stochastic load patterns, and equipment faults. Every decision trades cost, reliability, and equipment wear. Existing RL benchmarks either model grids deterministically or ignore multi-objective constraints. MicrogridEnv v2 fills this gap.

Why it is hard

• —Partial observability of the future — load and solar are noisy stochastic processes
• —Multi-objective trade-offs — cost, stability, and reliability interact non-trivially
• —Non-stationary events — storms and faults change the optimal policy mid-episode
• —Hard terminal conditions — blackout ends the episode immediately with a severe penalty
• —Credit assignment — component wear accumulates slowly; greedy policies look good short-term

State Space (12-dimensional)

Action Space (4-dimensional)

Example action:

{
  "battery_power": 3.0,
  "grid_import": 2.0,
  "load_shed_frac": 0.1,
  "dispatch_priority": 0
}

Meaning: discharge battery at 3 MW, import 2 MW from grid, shed 10% of load, prioritize solar first.

Reward Function

R = survival_bonus
    - alpha * operational_cost       (grid cost + battery wear cost)
    - beta  * stability_penalty      (quadratic: (|f - 50| - 0.2)^2)
    - gamma * outage_penalty         (unserved energy in MWh)
    - delta * shed_penalty           (load curtailment fraction)
    [- fault_penalty]                (only on cascade_fault task)

Weights are task-specific. See server/environment.py::_compute_reward.

Why this reward shape:

• —Quadratic stability term — forgiving near 50 Hz, punitive near the 1.5 Hz blackout boundary. Mirrors real power-system physics.
• —Survival bonus — prevents gaming the reward by triggering early termination.
• —Outage penalty dominates — reliability is always more important than cost.
• —Task-specific weights — steadystate rewards efficiency, cascadefault prioritizes survival.

Tasks

Clear progression in baseline scores confirms genuine difficulty differences.

Example Output (one episode)

Step  1 | freq=49.98 Hz  soc=0.38  reward=+0.140  fault=0
Step  2 | freq=50.03 Hz  soc=0.24  reward=+0.293  fault=0
Step  3 | freq=50.02 Hz  soc=0.12  reward=+0.295  fault=0
...
Step 24 | freq=49.99 Hz  soc=0.54  reward=+0.290  fault=0
total_reward = +2.72

Stochasticity

All randomness comes from a seedable numpy.random.RandomState:

• —Same seed = same trajectory (reproducibility)
• —Different seeds = genuinely different trajectories (research validity)
• —Load noise σ = variability × 3.0
• —Solar noise σ = variability × 5.0
• —Wind noise σ = variability × 4.0
• —Component degradation: random walk

Configurability

reset(task, seed, config) accepts an override dict to change task defaults without editing code:

env.reset(task="storm_day", seed=42, config={"variability": 0.4, "max_steps": 100})

Baseline Agents

• —DoNothingAgent — no control, lower bound
• —RandomAgent — uniform random actions
• —GreedyAgent — cost-minimizing heuristic (solar → battery → grid)
• —SmartHeuristicAgent — used inside grader for reproducible scoring

Baseline Performance

Real benchmark results — average over 5 seeds (1–5), reproducible with included baselines:

What this reveals about the environment:

• —Structure exists — GreedyAgent clearly outperforms Random on steadystate and stormday, confirming the environment rewards intelligent dispatch decisions.
• —Difficulty progression is real — all agents score lower as difficulty increases.
• —No single heuristic dominates — GreedyAgent collapses on cascade_fault (0.209) because it has no fault-response logic. Even RandomAgent (0.345) beats it there. This proves the hard task genuinely requires more than greedy reasoning — a clear opening for RL agents to learn superior strategies.
• —Room for RL improvement — the gap between the best heuristic and theoretical maximum (~1.0) is large, especially on cascade_fault.

Metrics Returned by Graders

Every grader call returns: score, freq_violations, blackout, total_unserved_mwh, total_grid_cost, difficulty, steps, rewards. Researchers can compare agents on multiple axes, not just a single score.

Quick Start

# 1. Install
pip install -e .

# 2. Run the quickstart demo (needs server running)
uvicorn server.app:app --host 0.0.0.0 --port 7860 &
python example.py

Runs all 3 tasks with the GreedyAgent in ~15 seconds.

Gymnasium Integration

Works with Stable-Baselines3, RLlib, CleanRL, or any library that consumes gymnasium.Env:

from microgrid_env_v2 import MicrogridGymEnv

env = MicrogridGymEnv(task="storm_day", seed=42)
obs, info = env.reset()
obs, reward, terminated, truncated, info = env.step(env.action_space.sample())

Docker

docker build -t microgrid-env-v2 .
docker run -d -p 7860:7860 microgrid-env-v2
curl http://localhost:7860/health

Tests

python tests/test_env.py   # core environment + grader variance
python tests/test_gym.py   # gymnasium wrapper

Both suites must pass before any deployment.

API Endpoints

Extensibility

Adding a new task requires only a new entry in TASK_CONFIGS and a scoring branch in grade_task. The physics module is separated from RL logic, making it easy to add:

• —Wind pricing
• —Demand response programs
• —Multi-agent coordination (multiple microgrids)
• —EV charging loads
• —Battery replacement decisions

Real-World Relevance

Similar optimization problems are faced by operators of real distributed energy systems worldwide — utilities, industrial microgrids, rural electrification projects, and campus microgrids. Current industry software typically uses rule-based SCADA control; RL policies trained on environments like this one represent the next generation of autonomous grid management.

For research: stochastic multi-objective control under terminal constraints is an open RL problem. This environment is a concrete testbed.

License

MIT