hanabhi/gridworld-env
0
1# OpenEnv: Production RL Made Simple2 3<div align="center">4 5<img src="https://upload.wikimedia.org/wikipedia/commons/1/10/PyTorch_logo_icon.svg" width="200" alt="PyTorch">6 7## From "Hello World" to RL Training in 5 Minutes ✨8 9**What if RL environments were as easy to use as REST APIs?**10 11That's OpenEnv. Type-safe. Isolated. Production-ready. 🎯12 13[](https://colab.research.google.com/github/meta-pytorch/OpenEnv/blob/main/examples/OpenEnv_Tutorial.ipynb)14[](https://github.com/meta-pytorch/OpenEnv)15[](https://opensource.org/licenses/BSD-3-Clause)16[](https://pytorch.org/)17 18Author: [Sanyam Bhutani](http://twitter.com/bhutanisanyam1/)19 20</div>21 22## Why OpenEnv?23 24Let's take a trip down memory lane:25 26It's 2016, RL is popular. You read some papers, it looks promising.27 28But in real world: Cartpole is the best you can run on a gaming GPU.29 30What do you do beyond Cartpole?31 32Fast-forward to 2025, GRPO is awesome and this time it's not JUST in theory, it works well in practise and is really here!33 34The problem still remains, how do you take these RL algorithms and take them beyond Cartpole?35 36A huge part of RL is giving your algorithms environment access to learn.37 38We are excited to introduce an Environment Spec for adding Open Environments for RL Training. This will allow you to focus on your experiments and allow everyone to bring their environments.39 40Focus on experiments, use OpenEnvironments, and build agents that go beyond Cartpole on a single spec.41 42---43 44## 📋 What You'll Learn45 46<table>47<tr>48<td width="50%">49 50**🎯 Part 1-2: The Fundamentals**51 52- ⚡ RL in 60 seconds53- 🤔 Why existing solutions fall short54- 💡 The OpenEnv solution55 56</td>57<td width="50%">58 59**🏗️ Part 3-5: The Architecture**60 61- 🔧 How OpenEnv works62- 🔍 Exploring real code63- 🎮 OpenSpiel integration example64 65</td>66</tr>67<tr>68<td width="50%">69 70**🎮 Part 6-8: Hands-On Demo**71 72- 🔌 Use existing OpenSpiel environment73- 🤖 Test 4 different policies74- 👀 Watch learning happen live75 76</td>77<td width="50%">78 79**🔧 Part 9-10: Going Further**80 81- 🎮 Switch to other OpenSpiel games82- ✨ Build your own integration83- 🌐 Deploy to production84 85</td>86</tr>87</table>88 89!!! tip "Pro Tip"90 This notebook is designed to run top-to-bottom in Google Colab with zero setup!91 92 ⏱️ **Time**: ~5 minutes | 📊 **Difficulty**: Beginner-friendly | 🎯 **Outcome**: Production-ready RL knowledge93 94---95 96## 📑 Table of Contents97 98### Foundation99 100- [Part 1: RL in 60 Seconds ⏱️](#part-1-rl-in-60-seconds)101- [Part 2: The Problem with Traditional RL 😤](#part-2-the-problem-with-traditional-rl)102- [Part 3: Setup 🛠️](#part-3-setup)103 104### Architecture105 106- [Part 4: The OpenEnv Pattern 🏗️](#part-4-the-openenv-pattern)107- [Part 5: Example Integration - OpenSpiel 🎮](#part-5-example-integration---openspiel)108 109### Hands-On Demo110 111- [Part 6: Interactive Demo 🎮](#part-6-using-real-openspiel)112- [Part 7: Four Policies 🤖](#part-7-four-policies)113- [Part 8: Policy Competition! 🏆](#part-8-policy-competition)114 115### Advanced116 117- [Part 9: Using Real OpenSpiel 🎮](#part-9-switching-to-other-games)118- [Part 10: Create Your Own Integration 🛠️](#part-10-create-your-own-integration)119 120### Wrap Up121 122- [Summary: Your Journey 🎓](#summary-your-journey)123- [Resources 📚](#resources)124 125---126 127(part-1-rl-in-60-seconds)=128## Part 1: RL in 60 Seconds ⏱️129 130**Reinforcement Learning is simpler than you think.**131 132It's just a loop:133 134```python135while not done:136 observation = environment.observe()137 action = policy.choose(observation)138 reward = environment.step(action)139 policy.learn(reward)140```141 142That's it. That's RL.143 144Let's see it in action:145 146```python147import random148 149print("🎲 " + "="*58 + " 🎲")150print(" Number Guessing Game - The Simplest RL Example")151print("🎲 " + "="*58 + " 🎲")152 153# Environment setup154target = random.randint(1, 10)155guesses_left = 3156 157print(f"\n🎯 I'm thinking of a number between 1 and 10...")158print(f"💭 You have {guesses_left} guesses. Let's see how random guessing works!\n")159 160# The RL Loop - Pure random policy (no learning!)161while guesses_left > 0:162 # Policy: Random guessing (no learning yet!)163 guess = random.randint(1, 10)164 guesses_left -= 1165 166 print(f"💭 Guess #{3-guesses_left}: {guess}", end=" → ")167 168 # Reward signal (but we're not using it!)169 if guess == target:170 print("🎉 Correct! +10 points")171 break172 elif abs(guess - target) <= 2:173 print("🔥 Warm! (close)")174 else:175 print("❄️ Cold! (far)")176else:177 print(f"\n💔 Out of guesses. The number was {target}.")178 179print("\n" + "="*62)180print("💡 This is RL: Observe → Act → Reward → Repeat")181print(" But this policy is terrible! It doesn't learn from rewards.")182print("="*62 + "\n")183```184 185**Output:**186```187🎲 ========================================================== 🎲188 Number Guessing Game - The Simplest RL Example189🎲 ========================================================== 🎲190 191🎯 I'm thinking of a number between 1 and 10...192💭 You have 3 guesses. Let's see how random guessing works!193 194💭 Guess #1: 2 → ❄️ Cold! (far)195💭 Guess #2: 10 → 🎉 Correct! +10 points196 197==============================================================198💡 This is RL: Observe → Act → Reward → Repeat199 But this policy is terrible! It doesn't learn from rewards.200==============================================================201```202 203---204 205(part-2-the-problem-with-traditional-rl)=206## Part 2: The Problem with Traditional RL 😤207 208### 🤔 Why Can't We Just Use OpenAI Gym?209 210Good question! Gym is great for research, but production needs more...211 212| Challenge | Traditional Approach | OpenEnv Solution |213|-----------|---------------------|------------------|214| **Type Safety** | ❌ `obs[0][3]` - what is this? | ✅ `obs.info_state` - IDE knows! |215| **Isolation** | ❌ Same process (can crash your training) | ✅ Docker containers (fully isolated) |216| **Deployment** | ❌ "Works on my machine" 🤷 | ✅ Same container everywhere 🐳 |217| **Scaling** | ❌ Hard to distribute | ✅ Deploy to Kubernetes ☸️ |218| **Language** | ❌ Python only | ✅ Any language (HTTP API) 🌐 |219| **Debugging** | ❌ Cryptic numpy errors | ✅ Clear type errors 🐛 |220 221### 💡 The OpenEnv Philosophy222 223**"RL environments should be like microservices"**224 225Think of it like this: You don't run your database in the same process as your web server, right? Same principle!226 227- 🔒 **Isolated**: Run in containers (security + stability)228- 🌐 **Standard**: HTTP API, works everywhere229- 📦 **Versioned**: Docker images (reproducibility!)230- 🚀 **Scalable**: Deploy to cloud with one command231- 🛡️ **Type-safe**: Catch bugs before they happen232- 🔄 **Portable**: Works on Mac, Linux, Windows, Cloud233 234### The Architecture235 236```237┌────────────────────────────────────────────────────────────┐238│ YOUR TRAINING CODE │239│ │240│ env = OpenSpielEnv(...) ← Import the client │241│ result = env.reset() ← Type-safe! │242│ result = env.step(action) ← Type-safe! │243│ │244└─────────────────┬──────────────────────────────────────────┘245 │246 │ HTTP/JSON (Language-Agnostic)247 │ POST /reset, POST /step, GET /state248 │249┌─────────────────▼──────────────────────────────────────────┐250│ DOCKER CONTAINER │251│ │252│ ┌──────────────────────────────────────────────┐ │253│ │ FastAPI Server │ │254│ │ └─ Environment (reset, step, state) │ │255│ │ └─ Your Game/Simulation Logic │ │256│ └──────────────────────────────────────────────┘ │257│ │258│ Isolated • Reproducible • Secure │259└────────────────────────────────────────────────────────────┘260```261 262!!! info "Key Insight"263 You never see HTTP details - just clean Python methods!264 265 ```python266 env.reset() # Under the hood: HTTP POST to /reset267 env.step(...) # Under the hood: HTTP POST to /step268 env.state() # Under the hood: HTTP GET to /state269 ```270 271 The magic? OpenEnv handles all the plumbing. You focus on RL! ✨272 273---274 275(part-3-setup)=276## Part 3: Setup 🛠️277 278**Running in Colab?** This cell will clone OpenEnv and install dependencies automatically.279 280**Running locally?** Make sure you're in the OpenEnv directory.281 282```ipython3283# Detect environment284try:285 import google.colab286 IN_COLAB = True287 print("🌐 Running in Google Colab - Perfect!")288except ImportError:289 IN_COLAB = False290 print("💻 Running locally - Nice!")291 292if IN_COLAB:293 print("\n📦 Cloning OpenEnv repository...")294 !git clone https://github.com/meta-pytorch/OpenEnv.git > /dev/null 2>&1295 %cd OpenEnv296 297 print("📚 Installing dependencies (this takes ~10 seconds)...")298 !pip install -q fastapi uvicorn requests299 300 import sys301 sys.path.insert(0, './src')302 print("\n✅ Setup complete! Everything is ready to go! 🎉")303else:304 import sys305 from pathlib import Path306 sys.path.insert(0, str(Path.cwd().parent / 'src'))307 print("✅ Using local OpenEnv installation")308 309print("\n🚀 Ready to explore OpenEnv and build amazing things!")310print("💡 Tip: Run cells top-to-bottom for the best experience.\n")311```312 313**Output:**314```315💻 Running locally - Nice!316✅ Using local OpenEnv installation317 318🚀 Ready to explore OpenEnv and build amazing things!319💡 Tip: Run cells top-to-bottom for the best experience.320```321 322---323 324(part-4-the-openenv-pattern)=325## Part 4: The OpenEnv Pattern 🏗️326 327### Every OpenEnv Environment Has 3 Components:328 329```330src/envs/your_env/331├── 📝 models.py ← Type-safe contracts332│ (Action, Observation, State)333│334├── 📱 client.py ← What YOU import335│ (HTTPEnvClient implementation)336│337└── 🖥️ server/338 ├── environment.py ← Game/simulation logic339 ├── app.py ← FastAPI server340 └── Dockerfile ← Container definition341```342 343Let's explore the actual OpenEnv code to see how this works:344 345```python346# Import OpenEnv's core abstractions347from core.env_server import Environment, Action, Observation, State348from core.http_env_client import HTTPEnvClient349 350print("="*70)351print(" 🧩 OPENENV CORE ABSTRACTIONS")352print("="*70)353 354print("""355🖥️ SERVER SIDE (runs in Docker):356 357 class Environment(ABC):358 '''Base class for all environment implementations'''359 360 @abstractmethod361 def reset(self) -> Observation:362 '''Start new episode'''363 364 @abstractmethod365 def step(self, action: Action) -> Observation:366 '''Execute action, return observation'''367 368 @property369 def state(self) -> State:370 '''Get episode metadata'''371 372📱 CLIENT SIDE (your training code):373 374 class HTTPEnvClient(ABC):375 '''Base class for HTTP clients'''376 377 def reset(self) -> StepResult:378 # HTTP POST /reset379 380 def step(self, action) -> StepResult:381 # HTTP POST /step382 383 def state(self) -> State:384 # HTTP GET /state385""")386 387print("="*70)388print("\n✨ Same interface on both sides - communication via HTTP!")389print("🎯 You focus on RL, OpenEnv handles the infrastructure.\n")390```391 392**Output:**393```394======================================================================395 🧩 OPENENV CORE ABSTRACTIONS396======================================================================397 398🖥️ SERVER SIDE (runs in Docker):399 400 class Environment(ABC):401 '''Base class for all environment implementations'''402 403 @abstractmethod404 def reset(self) -> Observation:405 '''Start new episode'''406 407 @abstractmethod408 def step(self, action: Action) -> Observation:409 '''Execute action, return observation'''410 411 @property412 def state(self) -> State:413 '''Get episode metadata'''414 415📱 CLIENT SIDE (your training code):416 417 class HTTPEnvClient(ABC):418 '''Base class for HTTP clients'''419 420 def reset(self) -> StepResult:421 # HTTP POST /reset422 423 def step(self, action) -> StepResult:424 # HTTP POST /step425 426 def state(self) -> State:427 # HTTP GET /state428 429======================================================================430 431✨ Same interface on both sides - communication via HTTP!432🎯 You focus on RL, OpenEnv handles the infrastructure.433```434 435---436 437(part-5-example-integration---openspiel)=438## Part 5: Example Integration - OpenSpiel 🎮439 440### What is OpenSpiel?441 442**OpenSpiel** is a library from DeepMind with **70+ game environments** for RL research.443 444### OpenEnv's Integration445 446We've wrapped **6 OpenSpiel games** following the OpenEnv pattern:447 448| **🎯 Single-Player** | **👥 Multi-Player** |449|---------------------|---------------------|450| 1. **Catch** - Catch falling ball | 5. **Tic-Tac-Toe** - Classic 3×3 |451| 2. **Cliff Walking** - Navigate grid | 6. **Kuhn Poker** - Imperfect info poker |452| 3. **2048** - Tile puzzle | |453| 4. **Blackjack** - Card game | |454 455This shows how OpenEnv can wrap **any** existing RL library!456 457```python458from envs.openspiel_env.client import OpenSpielEnv459 460print("="*70)461print(" 🔌 HOW OPENENV WRAPS OPENSPIEL")462print("="*70)463 464print("""465class OpenSpielEnv(HTTPEnvClient[OpenSpielAction, OpenSpielObservation]):466 467 def _step_payload(self, action: OpenSpielAction) -> dict:468 '''Convert typed action to JSON for HTTP'''469 return {470 "action_id": action.action_id,471 "game_name": action.game_name,472 }473 474 def _parse_result(self, payload: dict) -> StepResult:475 '''Parse HTTP JSON response into typed observation'''476 return StepResult(477 observation=OpenSpielObservation(...),478 reward=payload['reward'],479 done=payload['done']480 )481 482""")483 484print("─" * 70)485print("\n✨ Usage (works for ALL OpenEnv environments):")486print("""487 env = OpenSpielEnv(base_url="http://localhost:8000")488 489 result = env.reset()490 # Returns StepResult[OpenSpielObservation] - Type safe!491 492 result = env.step(OpenSpielAction(action_id=2, game_name="catch"))493 # Type checker knows this is valid!494 495 state = env.state()496 # Returns OpenSpielState497""")498 499print("─" * 70)500print("\n🎯 This pattern works for ANY environment you want to wrap!\n")501```502 503**Output:**504```505======================================================================506 🔌 HOW OPENENV WRAPS OPENSPIEL507======================================================================508 509class OpenSpielEnv(HTTPEnvClient[OpenSpielAction, OpenSpielObservation]):510 511 def _step_payload(self, action: OpenSpielAction) -> dict:512 '''Convert typed action to JSON for HTTP'''513 return {514 "action_id": action.action_id,515 "game_name": action.game_name,516 }517 518 def _parse_result(self, payload: dict) -> StepResult:519 '''Parse HTTP JSON response into typed observation'''520 return StepResult(521 observation=OpenSpielObservation(...),522 reward=payload['reward'],523 done=payload['done']524 )525 526 527──────────────────────────────────────────────────────────────────────528 529✨ Usage (works for ALL OpenEnv environments):530 531 env = OpenSpielEnv(base_url="http://localhost:8000")532 533 result = env.reset()534 # Returns StepResult[OpenSpielObservation] - Type safe!535 536 result = env.step(OpenSpielAction(action_id=2, game_name="catch"))537 # Type checker knows this is valid!538 539 state = env.state()540 # Returns OpenSpielState541 542──────────────────────────────────────────────────────────────────────543 544🎯 This pattern works for ANY environment you want to wrap!545```546 547### Type-Safe Models548 549```python550# Import OpenSpiel integration models551from envs.openspiel_env.models import (552 OpenSpielAction,553 OpenSpielObservation,554 OpenSpielState555)556from dataclasses import fields557 558print("="*70)559print(" 🎮 OPENSPIEL INTEGRATION - TYPE-SAFE MODELS")560print("="*70)561 562print("\n📤 OpenSpielAction (what you send):")563print(" " + "─" * 64)564for field in fields(OpenSpielAction):565 print(f" • {field.name:20s} : {field.type}")566 567print("\n📥 OpenSpielObservation (what you receive):")568print(" " + "─" * 64)569for field in fields(OpenSpielObservation):570 print(f" • {field.name:20s} : {field.type}")571 572print("\n📊 OpenSpielState (episode metadata):")573print(" " + "─" * 64)574for field in fields(OpenSpielState):575 print(f" • {field.name:20s} : {field.type}")576 577print("\n" + "="*70)578print("\n💡 Type safety means:")579print(" ✅ Your IDE autocompletes these fields")580print(" ✅ Typos are caught before running")581print(" ✅ Refactoring is safe")582print(" ✅ Self-documenting code\n")583```584 585**Output:**586```587======================================================================588 🎮 OPENSPIEL INTEGRATION - TYPE-SAFE MODELS589======================================================================590 591📤 OpenSpielAction (what you send):592 ────────────────────────────────────────────────────────────────593 • metadata : typing.Dict[str, typing.Any]594 • action_id : int595 • game_name : str596 • game_params : Dict[str, Any]597 598📥 OpenSpielObservation (what you receive):599 ────────────────────────────────────────────────────────────────600 • done : <class 'bool'>601 • reward : typing.Union[bool, int, float, NoneType]602 • metadata : typing.Dict[str, typing.Any]603 • info_state : List[float]604 • legal_actions : List[int]605 • game_phase : str606 • current_player_id : int607 • opponent_last_action : Optional[int]608 609📊 OpenSpielState (episode metadata):610 ────────────────────────────────────────────────────────────────611 • episode_id : typing.Optional[str]612 • step_count : <class 'int'>613 • game_name : str614 • agent_player : int615 • opponent_policy : str616 • game_params : Dict[str, Any]617 • num_players : int618 619======================================================================620 621💡 Type safety means:622 ✅ Your IDE autocompletes these fields623 ✅ Typos are caught before running624 ✅ Refactoring is safe625 ✅ Self-documenting code626```627 628### How the Client Works629 630The client **inherits from HTTPEnvClient** and implements 3 methods:631 6321. `_step_payload()` - Convert action → JSON6332. `_parse_result()` - Parse JSON → typed observation6343. `_parse_state()` - Parse JSON → state635 636That's it! The base class handles all HTTP communication.637 638---639 640(part-6-using-real-openspiel)=641## Part 6: Using Real OpenSpiel 🎮642 643<div style="text-align: center; background: linear-gradient(135deg, #667eea 0%, #764ba2 100%); color: white; padding: 30px; border-radius: 15px; margin: 30px 0;">644 645### Now let's USE a production environment!646 647We'll play **Catch** using OpenEnv's **OpenSpiel integration** 🎯648 649This is a REAL environment running in production at companies!650 651**Get ready for:**652 653- 🔌 Using existing environments (not building)654- 🤖 Testing policies against real games655- 📊 Live gameplay visualization656- 🎯 Production-ready patterns657 658</div>659 660### The Game: Catch 🔴🏓661 662```663⬜ ⬜ 🔴 ⬜ ⬜664⬜ ⬜ ⬜ ⬜ ⬜665⬜ ⬜ ⬜ ⬜ ⬜ Ball666⬜ ⬜ ⬜ ⬜ ⬜667⬜ ⬜ ⬜ ⬜ ⬜ falls668⬜ ⬜ ⬜ ⬜ ⬜669⬜ ⬜ ⬜ ⬜ ⬜ down670⬜ ⬜ ⬜ ⬜ ⬜671⬜ ⬜ ⬜ ⬜ ⬜672⬜ ⬜ 🏓 ⬜ ⬜673 Paddle674```675 676**Rules:**677 678- 10×5 grid679- Ball falls from random column680- Move paddle left/right to catch it681 682**Actions:**683 684- `0` = Move LEFT ⬅️685- `1` = STAY 🛑686- `2` = Move RIGHT ➡️687 688**Reward:**689 690- `+1` if caught 🎉691- `0` if missed 😢692 693!!! note "Why Catch?"694 - Simple rules (easy to understand)695 - Fast episodes (~5 steps)696 - Clear success/failure697 - Part of OpenSpiel's 70+ games!698 699 **💡 The Big Idea:**700 Instead of building this from scratch, we'll USE OpenEnv's existing OpenSpiel integration. Same interface, but production-ready!701 702```python703from envs.openspiel_env import OpenSpielEnv704from envs.openspiel_env.models import (705 OpenSpielAction,706 OpenSpielObservation,707 OpenSpielState708)709from dataclasses import fields710 711print("🎮 " + "="*64 + " 🎮")712print(" ✅ Importing Real OpenSpiel Environment!")713print("🎮 " + "="*64 + " 🎮\n")714 715print("📦 What we just imported:")716print(" • OpenSpielEnv - HTTP client for OpenSpiel games")717print(" • OpenSpielAction - Type-safe actions")718print(" • OpenSpielObservation - Type-safe observations")719print(" • OpenSpielState - Episode metadata\n")720 721print("📋 OpenSpielObservation fields:")722print(" " + "─" * 60)723for field in fields(OpenSpielObservation):724 print(f" • {field.name:25s} : {field.type}")725 726print("\n" + "="*70)727print("\n💡 This is REAL OpenEnv code - used in production!")728print(" • Wraps 6 OpenSpiel games (Catch, Tic-Tac-Toe, Poker, etc.)")729print(" • Type-safe actions and observations")730print(" • Works via HTTP (we'll see that next!)\n")731```732 733**Output:**734```735🎮 ================================================================ 🎮736 ✅ Importing Real OpenSpiel Environment!737🎮 ================================================================ 🎮738 739📦 What we just imported:740 • OpenSpielEnv - HTTP client for OpenSpiel games741 • OpenSpielAction - Type-safe actions742 • OpenSpielObservation - Type-safe observations743 • OpenSpielState - Episode metadata744 745📋 OpenSpielObservation fields:746 ────────────────────────────────────────────────────────────747 • done : <class 'bool'>748 • reward : typing.Union[bool, int, float, NoneType]749 • metadata : typing.Dict[str, typing.Any]750 • info_state : List[float]751 • legal_actions : List[int]752 • game_phase : str753 • current_player_id : int754 • opponent_last_action : Optional[int]755 756======================================================================757 758💡 This is REAL OpenEnv code - used in production!759 • Wraps 6 OpenSpiel games (Catch, Tic-Tac-Toe, Poker, etc.)760 • Type-safe actions and observations761 • Works via HTTP (we'll see that next!)762```763 764---765 766(part-7-four-policies)=767## Part 7: Four Policies 🤖768 769Let's test 4 different AI strategies:770 771| Policy | Strategy | Expected Performance |772|--------|----------|----------------------|773| **🎲 Random** | Pick random action every step | ~20% (pure luck) |774| **🛑 Always Stay** | Never move, hope ball lands in center | ~20% (terrible!) |775| **🧠 Smart** | Move paddle toward ball | 100% (optimal!) |776| **📈 Learning** | Start random, learn smart strategy | ~85% (improves over time) |777 778**💡 These policies work with ANY OpenSpiel game!**779 780```python781import random782 783# ============================================================================784# POLICIES - Different AI strategies (adapted for OpenSpiel)785# ============================================================================786 787class RandomPolicy:788 """Baseline: Pure random guessing."""789 name = "🎲 Random Guesser"790 791 def select_action(self, obs: OpenSpielObservation) -> int:792 return random.choice(obs.legal_actions)793 794 795class AlwaysStayPolicy:796 """Bad strategy: Never moves."""797 name = "🛑 Always Stay"798 799 def select_action(self, obs: OpenSpielObservation) -> int:800 return 1 # STAY801 802 803class SmartPolicy:804 """Optimal: Move paddle toward ball."""805 name = "🧠 Smart Heuristic"806 807 def select_action(self, obs: OpenSpielObservation) -> int:808 # Parse OpenSpiel observation809 # For Catch: info_state is a flattened 10x5 grid810 # Ball position and paddle position encoded in the vector811 info_state = obs.info_state812 813 # Find ball and paddle positions from info_state814 # Catch uses a 10x5 grid, so 50 values815 grid_size = 5816 817 # Find positions (ball = 1.0 in the flattened grid, paddle = 1.0 in the last row of the flattened grid)818 ball_col = None819 paddle_col = None820 821 for idx, val in enumerate(info_state):822 if abs(val - 1.0) < 0.01: # Ball823 ball_col = idx % grid_size824 break825 826 last_row = info_state[-grid_size:]827 paddle_col = last_row.index(1.0) # Paddle828 829 if ball_col is not None and paddle_col is not None:830 if paddle_col < ball_col:831 return 2 # Move RIGHT832 elif paddle_col > ball_col:833 return 0 # Move LEFT834 835 return 1 # STAY (fallback)836 837 838class LearningPolicy:839 """Simulated RL: Epsilon-greedy exploration."""840 name = "📈 Learning Agent"841 842 def __init__(self):843 self.steps = 0844 self.smart_policy = SmartPolicy()845 846 def select_action(self, obs: OpenSpielObservation) -> int:847 self.steps += 1848 849 # Decay exploration rate over time850 epsilon = max(0.1, 1.0 - (self.steps / 100))851 852 if random.random() < epsilon:853 # Explore: random action854 return random.choice(obs.legal_actions)855 else:856 # Exploit: use smart strategy857 return self.smart_policy.select_action(obs)858 859 860print("🤖 " + "="*64 + " 🤖")861print(" ✅ 4 Policies Created (Adapted for OpenSpiel)!")862print("🤖 " + "="*64 + " 🤖\n")863 864policies = [RandomPolicy(), AlwaysStayPolicy(), SmartPolicy(), LearningPolicy()]865for i, policy in enumerate(policies, 1):866 print(f" {i}. {policy.name}")867 868print("\n💡 These policies work with OpenSpielObservation!")869print(" • Read info_state (flattened grid)")870print(" • Use legal_actions")871print(" • Work with ANY OpenSpiel game that exposes these!\n")872```873 874**Output:**875```876🤖 ================================================================ 🤖877 ✅ 4 Policies Created (Adapted for OpenSpiel)!878🤖 ================================================================ 🤖879 880 1. 🎲 Random Guesser881 2. 🛑 Always Stay882 3. 🧠 Smart Heuristic883 4. 📈 Learning Agent884 885💡 These policies work with OpenSpielObservation!886 • Read info_state (flattened grid)887 • Use legal_actions888 • Work with ANY OpenSpiel game that exposes these!889```890 891---892 893(part-8-policy-competition)=894## Part 8: Policy Competition! 🏆895 896Let's run **50 episodes** for each policy against **REAL OpenSpiel** and see who wins!897 898This is production code - every action is an HTTP call to the OpenSpiel server!899 900```python901def evaluate_policies(env, num_episodes=50):902 """Compare all policies over many episodes using real OpenSpiel."""903 policies = [904 RandomPolicy(),905 AlwaysStayPolicy(),906 SmartPolicy(),907 LearningPolicy(),908 ]909 910 print("\n🏆 " + "="*66 + " 🏆")911 print(f" POLICY SHOWDOWN - {num_episodes} Episodes Each")912 print(f" Playing against REAL OpenSpiel Catch!")913 print("🏆 " + "="*66 + " 🏆\n")914 915 results = []916 for policy in policies:917 print(f"⚡ Testing {policy.name}...", end=" ")918 successes = sum(run_episode(env, policy, visualize=False)919 for _ in range(num_episodes))920 success_rate = (successes / num_episodes) * 100921 results.append((policy.name, success_rate, successes))922 print(f"✓ Done!")923 924 print("\n" + "="*70)925 print(" 📊 FINAL RESULTS")926 print("="*70 + "\n")927 928 # Sort by success rate (descending)929 results.sort(key=lambda x: x[1], reverse=True)930 931 # Award medals to top 3932 medals = ["🥇", "🥈", "🥉", " "]933 934 for i, (name, rate, successes) in enumerate(results):935 medal = medals[i]936 bar = "█" * int(rate / 2)937 print(f"{medal} {name:25s} [{bar:<50}] {rate:5.1f}% ({successes}/{num_episodes})")938 939 print("\n" + "="*70)940 print("\n✨ Key Insights:")941 print(" • Random (~20%): Baseline - pure luck 🎲")942 print(" • Always Stay (~20%): Bad strategy - stays center 🛑")943 print(" • Smart (100%): Optimal - perfect play! 🧠")944 print(" • Learning (~85%): Improves over time 📈")945 print("\n🎓 This is Reinforcement Learning + OpenEnv in action:")946 print(" 1. We USED existing OpenSpiel environment (didn't build it)")947 print(" 2. Type-safe communication over HTTP")948 print(" 3. Same code works for ANY OpenSpiel game")949 print(" 4. Production-ready architecture\n")950 951# Run the epic competition!952print("🎮 Starting the showdown against REAL OpenSpiel...\n")953evaluate_policies(client, num_episodes=50)954```955 956---957 958(part-9-switching-to-other-games)=959## Part 9: Switching to Other Games 🎮960 961### What We Just Used: Real OpenSpiel! 🎉962 963In Parts 6-8, we **USED** the existing OpenSpiel Catch environment:964 965| What We Did | How It Works |966|-------------|--------------|967| **Imported** | OpenSpielEnv client (pre-built) |968| **Started** | OpenSpiel server via uvicorn |969| **Connected** | HTTP client to server |970| **Played** | Real OpenSpiel Catch game |971 972**🎯 This is production code!** Every action was an HTTP call to a real OpenSpiel environment.973 974### 🎮 6 Games Available - Same Interface!975 976The beauty of OpenEnv? **Same code, different games!**977 978```python979# We just used Catch980env = OpenSpielEnv(base_url="http://localhost:8000")981# game_name="catch" was set via environment variable982 983# Want Tic-Tac-Toe instead? Just change the game!984# Start server with: OPENSPIEL_GAME=tic_tac_toe uvicorn ...985# Same client code works!986```987 988**🎮 All 6 Games:**989 9901. ✅ **`catch`** - What we just used!9912. **`tic_tac_toe`** - Classic 3×39923. **`kuhn_poker`** - Imperfect information poker9934. **`cliff_walking`** - Grid navigation9945. **`2048`** - Tile puzzle9956. **`blackjack`** - Card game996 997**All use the exact same OpenSpielEnv client!**998 999### Try Another Game (Optional):1000 1001```python1002# Stop the current server (kill the server_process)1003# Then start a new game:1004 1005server_process = subprocess.Popen(1006 [sys.executable, "-m", "uvicorn",1007 "envs.openspiel_env.server.app:app",1008 "--host", "0.0.0.0",1009 "--port", "8000"],1010 env={**os.environ,1011 "PYTHONPATH": f"{work_dir}/src",1012 "OPENSPIEL_GAME": "tic_tac_toe", # Changed!1013 "OPENSPIEL_AGENT_PLAYER": "0",1014 "OPENSPIEL_OPPONENT_POLICY": "random"},1015 # ... rest of config1016)1017 1018# Same client works!1019client = OpenSpielEnv(base_url="http://localhost:8000")1020result = client.reset() # Now playing Tic-Tac-Toe!1021```1022 1023**💡 Key Insight**: You don't rebuild anything - you just USE different games with the same client!1024 1025---1026 1027(part-10-create-your-own-integration)=1028## Part 10: Create Your Own Integration 🛠️1029 1030### The 5-Step Pattern1031 1032Want to wrap your own environment in OpenEnv? Here's how:1033 1034### Step 1: Define Types (`models.py`)1035 1036```python1037from dataclasses import dataclass1038from core.env_server import Action, Observation, State1039 1040@dataclass1041class YourAction(Action):1042 action_value: int1043 # Add your action fields1044 1045@dataclass1046class YourObservation(Observation):1047 state_data: List[float]1048 done: bool1049 reward: float1050 # Add your observation fields1051 1052@dataclass1053class YourState(State):1054 episode_id: str1055 step_count: int1056 # Add your state fields1057```1058 1059### Step 2: Implement Environment (`server/environment.py`)1060 1061```python1062from core.env_server import Environment1063 1064class YourEnvironment(Environment):1065 def reset(self) -> Observation:1066 # Initialize your game/simulation1067 return YourObservation(...)1068 1069 def step(self, action: Action) -> Observation:1070 # Execute action, update state1071 return YourObservation(...)1072 1073 @property1074 def state(self) -> State:1075 return self._state1076```1077 1078### Step 3: Create Client (`client.py`)1079 1080```python1081from core.http_env_client import HTTPEnvClient1082from core.types import StepResult1083 1084class YourEnv(HTTPEnvClient[YourAction, YourObservation]):1085 def _step_payload(self, action: YourAction) -> dict:1086 """Convert action to JSON"""1087 return {"action_value": action.action_value}1088 1089 def _parse_result(self, payload: dict) -> StepResult:1090 """Parse JSON to observation"""1091 return StepResult(1092 observation=YourObservation(...),1093 reward=payload['reward'],1094 done=payload['done']1095 )1096 1097 def _parse_state(self, payload: dict) -> YourState:1098 return YourState(...)1099```1100 1101### Step 4: Create Server (`server/app.py`)1102 1103```python1104from core.env_server import create_fastapi_app1105from .your_environment import YourEnvironment1106 1107env = YourEnvironment()1108app = create_fastapi_app(env)1109 1110# That's it! OpenEnv creates all endpoints for you.1111```1112 1113### Step 5: Dockerize (`server/Dockerfile`)1114 1115```dockerfile1116FROM python:3.11-slim1117 1118WORKDIR /app1119COPY requirements.txt .1120RUN pip install --no-cache-dir -r requirements.txt1121 1122COPY . .1123CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]1124```1125 1126### 🎓 Examples to Study1127 1128OpenEnv includes 3 complete examples:1129 11301. **`src/envs/echo_env/`**1131 - Simplest possible environment1132 - Great for testing and learning1133 11342. **`src/envs/openspiel_env/`**1135 - Wraps external library (OpenSpiel)1136 - Shows integration pattern1137 - 6 games in one integration1138 11393. **`src/envs/coding_env/`**1140 - Python code execution environment1141 - Shows complex use case1142 - Security considerations1143 1144**💡 Study these to understand the patterns!**1145 1146---1147 1148(summary-your-journey)=1149## 🎓 Summary: Your Journey1150 1151### What You Learned1152 1153<table>1154<tr>1155<td width="50%" style="vertical-align: top;">1156 1157### 📚 Concepts1158 1159✅ **RL Fundamentals**1160 1161- The observe-act-reward loop1162- What makes good policies1163- Exploration vs exploitation1164 1165✅ **OpenEnv Architecture**1166 1167- Client-server separation1168- Type-safe contracts1169- HTTP communication layer1170 1171✅ **Production Patterns**1172 1173- Docker isolation1174- API design1175- Reproducible deployments1176 1177</td>1178<td width="50%" style="vertical-align: top;">1179 1180### 🛠️ Skills1181 1182✅ **Using Environments**1183 1184- Import OpenEnv clients1185- Call reset/step/state1186- Work with typed observations1187 1188✅ **Building Environments**1189 1190- Define type-safe models1191- Implement Environment class1192- Create HTTPEnvClient1193 1194✅ **Testing & Debugging**1195 1196- Compare policies1197- Visualize episodes1198- Measure performance1199 1200</td>