hanabhi/gridworld-env
0
1{2 "cells": [3 {4 "cell_type": "markdown",5 "metadata": {},6 "source": [7 "<div align=\"center\">\n",8 "\n",9 "<img src=\"https://upload.wikimedia.org/wikipedia/commons/1/10/PyTorch_logo_icon.svg\" width=\"200\" alt=\"PyTorch\">\n",10 "\n",11 "\n",12 "\n",13 "# OpenEnv: Production RL Made Simple\n",14 "\n",15 "### *From \"Hello World\" to RL Training in 5 Minutes* ✨\n",16 "\n",17 "---\n",18 "\n",19 "**What if RL environments were as easy to use as REST APIs?**\n",20 "\n",21 "That's OpenEnv. Type-safe. Isolated. Production-ready. 🎯\n",22 "\n",23 "[](https://github.com/meta-pytorch/OpenEnv)\n",24 "[](https://opensource.org/licenses/BSD-3-Clause)\n",25 "[](https://pytorch.org/)\n",26 "\n",27 "Author: [Sanyam Bhutani](http://twitter.com/bhutanisanyam1/)\n",28 "\n",29 "</div>\n",30 "\n",31 "\n",32 "\n",33 "---"34 ]35 },36 {37 "cell_type": "markdown",38 "metadata": {},39 "source": [40 "---\n",41 "\n",42 "## Why OpenEnv?\n",43 "\n",44 "Let's take a trip down memory lane:\n",45 "\n",46 "It's 2016, RL is popular. You read some papers, it looks promising. \n",47 "\n",48 "But in real world: Cartpole is the best you can run on a gaming GPU. \n",49 "\n",50 "What do you do beyond Cartpole?\n",51 "\n",52 "Fast-forward to 2025, GRPO is awesome and this time it's not JUST in theory, it works well in practise and is really here!\n",53 "\n",54 "The problem still remains, how do you take these RL algorithms and take them beyond Cartpole?\n",55 "\n",56 "A huge part of RL is giving your algorithms environment access to learn. \n",57 "\n",58 "We are excited to introduce an Environment Spec for adding Open Environments for RL Training. This will allow you to focus on your experiments and allow everyone to bring their environments.\n",59 "\n",60 "Focus on experiments, use OpenEnvironments, and build agents that go beyond Cartpole on a single spec.\n",61 "\n",62 "---"63 ]64 },65 {66 "cell_type": "markdown",67 "metadata": {},68 "source": [69 "## 📋 What You'll Learn\n",70 "\n",71 "<table>\n",72 "<tr>\n",73 "<td width=\"50%\">\n",74 "\n",75 "**🎯 Part 1-2: The Fundamentals**\n",76 "- ⚡ RL in 60 seconds\n",77 "- 🤔 Why existing solutions fall short\n",78 "- 💡 The OpenEnv solution\n",79 "\n",80 "</td>\n",81 "<td width=\"50%\">\n",82 "\n",83 "**🏗️ Part 3-5: The Architecture**\n",84 "- 🔧 How OpenEnv works\n",85 "- 🔍 Exploring real code\n",86 "- 🎮 OpenSpiel integration example\n",87 "\n",88 "</td>\n",89 "</tr>\n",90 "<tr>\n",91 "<td width=\"50%\">\n",92 "\n",93 "**🎮 Part 6-8: Hands-On Demo**\n",94 "- 🔌 Use existing OpenSpiel environment\n",95 "- 🤖 Test 4 different policies\n",96 "- 👀 Watch learning happen live\n",97 "\n",98 "</td>\n",99 "<td width=\"50%\">\n",100 "\n",101 "**🔧 Part 9-10: Going Further**\n",102 "- 🎮 Switch to other OpenSpiel games\n",103 "- ✨ Build your own integration\n",104 "- 🌐 Deploy to production\n",105 "\n",106 "</td>\n",107 "</tr>\n",108 "</table>\n",109 "\n",110 "> 💡 **Pro Tip**: This notebook is designed to run top-to-bottom in Google Colab with zero setup!\n",111 ">\n",112 "> ⏱️ **Time**: ~5 minutes | 📊 **Difficulty**: Beginner-friendly | 🎯 **Outcome**: Production-ready RL knowledge\n"113 ]114 },115 {116 "cell_type": "markdown",117 "metadata": {},118 "source": [119 "---\n",120 "\n",121 "## 📑 Table of Contents\n",122 "\n",123 "<div style=\"background-color: rgba(128, 128, 128, 0.1); padding: 20px; border-radius: 10px; margin: 20px 0; border: 1px solid rgba(128, 128, 128, 0.2);\">\n",124 "\n",125 "**Quick Navigation** - Click any section to jump right there! 🎯\n",126 "\n",127 "### Foundation\n",128 "- [Part 1: RL in 60 Seconds ⏱️](#part-1)\n",129 "- [Part 2: The Problem with Traditional RL 😤](#part-2)\n",130 "- [Part 3: Setup 🛠️](#part-3)\n",131 "\n",132 "### Architecture\n",133 "- [Part 4: The OpenEnv Pattern 🏗️](#part-4)\n",134 "- [Part 5: Example Integration - OpenSpiel 🎮](#part-5)\n",135 "\n",136 "### Hands-On Demo\n",137 "- [Part 6: Interactive Demo 🎮](#part-6)\n",138 "- [Part 7: Four Policies 🤖](#part-7)\n",139 "- [Part 8: Policy Competition! 🏆](#part-8)\n",140 "\n",141 "### Advanced\n",142 "- [Part 9: Using Real OpenSpiel 🎮](#part-9)\n",143 "- [Part 10: Create Your Own Integration 🛠️](#part-10)\n",144 "\n",145 "### Wrap Up\n",146 "- [Summary: Your Journey 🎓](#summary)\n",147 "- [Resources 📚](#resources)\n",148 "\n",149 "</div>\n",150 "\n",151 "---"152 ]153 },154 {155 "cell_type": "markdown",156 "metadata": {},157 "source": [158 "---\n",159 "\n",160 "# Part 1: RL in 60 Seconds ⏱️\n",161 "\n",162 "<div style=\"background-color: rgba(128, 128, 128, 0.1); padding: 20px; border-radius: 10px; margin: 20px 0; border: 1px solid rgba(128, 128, 128, 0.2);\">\n",163 "\n",164 "**Reinforcement Learning is simpler than you think.**\n",165 "\n",166 "It's just a loop:\n",167 "\n",168 "```python\n",169 "while not done:\n",170 " observation = environment.observe()\n",171 " action = policy.choose(observation)\n",172 " reward = environment.step(action)\n",173 " policy.learn(reward)\n",174 "```\n",175 "\n",176 "That's it. That's RL.\n",177 "\n",178 "</div>\n",179 "\n",180 "Let's see it in action:"181 ]182 },183 {184 "cell_type": "code",185 "execution_count": 1,186 "metadata": {},187 "outputs": [188 {189 "name": "stdout",190 "output_type": "stream",191 "text": [192 "🎲 ========================================================== 🎲\n",193 " Number Guessing Game - The Simplest RL Example\n",194 "🎲 ========================================================== 🎲\n",195 "\n",196 "🎯 I'm thinking of a number between 1 and 10...\n",197 "💭 You have 3 guesses. Let's see how random guessing works!\n",198 "\n",199 "💭 Guess #1: 4 → ❄️ Cold! (far)\n",200 "💭 Guess #2: 4 → ❄️ Cold! (far)\n",201 "💭 Guess #3: 3 → ❄️ Cold! (far)\n",202 "\n",203 "💔 Out of guesses. The number was 8.\n",204 "\n",205 "==============================================================\n",206 "💡 This is RL: Observe → Act → Reward → Repeat\n",207 " But this policy is terrible! It doesn't learn from rewards.\n",208 "==============================================================\n",209 "\n"210 ]211 }212 ],213 "source": [214 "import random\n",215 "\n",216 "print(\"🎲 \" + \"=\"*58 + \" 🎲\")\n",217 "print(\" Number Guessing Game - The Simplest RL Example\")\n",218 "print(\"🎲 \" + \"=\"*58 + \" 🎲\")\n",219 "\n",220 "# Environment setup\n",221 "target = random.randint(1, 10)\n",222 "guesses_left = 3\n",223 "\n",224 "print(f\"\\n🎯 I'm thinking of a number between 1 and 10...\")\n",225 "print(f\"💭 You have {guesses_left} guesses. Let's see how random guessing works!\\n\")\n",226 "\n",227 "# The RL Loop - Pure random policy (no learning!)\n",228 "while guesses_left > 0:\n",229 " # Policy: Random guessing (no learning yet!)\n",230 " guess = random.randint(1, 10)\n",231 " guesses_left -= 1\n",232 " \n",233 " print(f\"💭 Guess #{3-guesses_left}: {guess}\", end=\" → \")\n",234 " \n",235 " # Reward signal (but we're not using it!)\n",236 " if guess == target:\n",237 " print(\"🎉 Correct! +10 points\")\n",238 " break\n",239 " elif abs(guess - target) <= 2:\n",240 " print(\"🔥 Warm! (close)\")\n",241 " else:\n",242 " print(\"❄️ Cold! (far)\")\n",243 "else:\n",244 " print(f\"\\n💔 Out of guesses. The number was {target}.\")\n",245 "\n",246 "print(\"\\n\" + \"=\"*62)\n",247 "print(\"💡 This is RL: Observe → Act → Reward → Repeat\")\n",248 "print(\" But this policy is terrible! It doesn't learn from rewards.\")\n",249 "print(\"=\"*62 + \"\\n\")"250 ]251 },252 {253 "cell_type": "markdown",254 "metadata": {},255 "source": [256 "---\n",257 "\n",258 "<a id=\"part-2\"></a>\n",259 "# Part 2: The Problem with Traditional RL 😤\n",260 "\n",261 "<div style=\"background-color: rgba(248, 205, 16, 0.1); padding: 20px; border-radius: 10px; margin: 20px 0; border: 1px solid rgba(248, 205, 16, 0.1);\">\n",262 "\n",263 "## 🤔 Why Can't We Just Use OpenAI Gym?\n",264 "\n",265 "Good question! Gym is great for research, but production needs more...\n",266 "\n",267 "</div>\n",268 "\n",269 "<table>\n",270 "<tr>\n",271 "<th>Challenge</th>\n",272 "<th>Traditional Approach</th>\n",273 "<th>OpenEnv Solution</th>\n",274 "</tr>\n",275 "<tr>\n",276 "<td><b>Type Safety</b></td>\n",277 "<td>❌ <code>obs[0][3]</code> - what is this?</td>\n",278 "<td>✅ <code>obs.info_state</code> - IDE knows!</td>\n",279 "</tr>\n",280 "<tr>\n",281 "<td><b>Isolation</b></td>\n",282 "<td>❌ Same process (can crash your training)</td>\n",283 "<td>✅ Docker containers (fully isolated)</td>\n",284 "</tr>\n",285 "<tr>\n",286 "<td><b>Deployment</b></td>\n",287 "<td>❌ \"Works on my machine\" 🤷</td>\n",288 "<td>✅ Same container everywhere 🐳</td>\n",289 "</tr>\n",290 "<tr>\n",291 "<td><b>Scaling</b></td>\n",292 "<td>❌ Hard to distribute</td>\n",293 "<td>✅ Deploy to Kubernetes ☸️</td>\n",294 "</tr>\n",295 "<tr>\n",296 "<td><b>Language</b></td>\n",297 "<td>❌ Python only</td>\n",298 "<td>✅ Any language (HTTP API) 🌐</td>\n",299 "</tr>\n",300 "<tr>\n",301 "<td><b>Debugging</b></td>\n",302 "<td>❌ Cryptic numpy errors</td>\n",303 "<td>✅ Clear type errors 🐛</td>\n",304 "</tr>\n",305 "</table>\n",306 "\n",307 "<div style=\"background-color: #d4edda; padding: 20px; border-left: 5px solid #28a745; margin: 20px 0;\">\n",308 "\n",309 "## 💡 The OpenEnv Philosophy\n",310 "\n",311 "**\"RL environments should be like microservices\"**\n",312 "\n",313 "Think of it like this: You don't run your database in the same process as your web server, right? Same principle!\n",314 "\n",315 "- 🔒 **Isolated**: Run in containers (security + stability)\n",316 "- 🌐 **Standard**: HTTP API, works everywhere\n",317 "- 📦 **Versioned**: Docker images (reproducibility!)\n",318 "- 🚀 **Scalable**: Deploy to cloud with one command\n",319 "- 🛡️ **Type-safe**: Catch bugs before they happen\n",320 "- 🔄 **Portable**: Works on Mac, Linux, Windows, Cloud\n",321 "\n",322 "</div>"323 ]324 },325 {326 "cell_type": "markdown",327 "metadata": {},328 "source": [329 "### The Architecture\n",330 "\n",331 "```\n",332 "┌────────────────────────────────────────────────────────────┐\n",333 "│ YOUR TRAINING CODE │\n",334 "│ │\n",335 "│ env = OpenSpielEnv(...) ← Import the client │\n",336 "│ result = env.reset() ← Type-safe! │\n",337 "│ result = env.step(action) ← Type-safe! │\n",338 "│ │\n",339 "└─────────────────┬──────────────────────────────────────────┘\n",340 " │\n",341 " │ HTTP/JSON (Language-Agnostic)\n",342 " │ POST /reset, POST /step, GET /state\n",343 " │\n",344 "┌─────────────────▼──────────────────────────────────────────┐\n",345 "│ DOCKER CONTAINER │\n",346 "│ │\n",347 "│ ┌──────────────────────────────────────────────┐ │\n",348 "│ │ FastAPI Server │ │\n",349 "│ │ └─ Environment (reset, step, state) │ │\n",350 "│ │ └─ Your Game/Simulation Logic │ │\n",351 "│ └──────────────────────────────────────────────┘ │\n",352 "│ │\n",353 "│ Isolated • Reproducible • Secure │\n",354 "└────────────────────────────────────────────────────────────┘\n",355 "```\n",356 "\n",357 "<div style=\"background-color: #e7f3ff; padding: 15px; border-left: 5px solid #0366d6; margin: 20px 0;\">\n",358 "\n",359 "**🎯 Key Insight**: You never see HTTP details - just clean Python methods!\n",360 "\n",361 "```python\n",362 "env.reset() # Under the hood: HTTP POST to /reset\n",363 "env.step(...) # Under the hood: HTTP POST to /step\n",364 "env.state() # Under the hood: HTTP GET to /state\n",365 "```\n",366 "\n",367 "The magic? OpenEnv handles all the plumbing. You focus on RL! ✨\n",368 "\n",369 "</div>"370 ]371 },372 {373 "cell_type": "markdown",374 "metadata": {},375 "source": [376 "---\n",377 "\n",378 "# Part 3: Setup 🛠️\n",379 "\n",380 "<div style=\"background-color: #f8f9fa; padding: 15px; border-radius: 5px; margin: 20px 0;\">\n",381 "\n",382 "**Running in Colab?** This cell will clone OpenEnv and install dependencies automatically.\n",383 "\n",384 "**Running locally?** Make sure you're in the OpenEnv directory.\n",385 "\n",386 "</div>"387 ]388 },389 {390 "cell_type": "code",391 "execution_count": null,392 "metadata": {},393 "outputs": [394 {395 "name": "stdout",396 "output_type": "stream",397 "text": [398 "💻 Running locally - Nice!\n",399 "✅ Using local OpenEnv installation\n",400 "\n",401 "🚀 Ready to explore OpenEnv and build amazing things!\n",402 "💡 Tip: Run cells top-to-bottom for the best experience.\n",403 "\n"404 ]405 }406 ],407 "source": [408 "# Detect environment\n",409 "try:\n",410 " import google.colab\n",411 " IN_COLAB = True\n",412 " print(\"🌐 Running in Google Colab - Perfect!\")\n",413 "except ImportError:\n",414 " IN_COLAB = False\n",415 " print(\"💻 Running locally - Nice!\")\n",416 "\n",417 "if IN_COLAB:\n",418 " print(\"\\n📦 Cloning OpenEnv repository...\")\n",419 " !git clone https://github.com/meta-pytorch/OpenEnv.git > /dev/null 2>&1\n",420 " %cd OpenEnv\n",421 " !git checkout c2997e8 > /dev/null 2>&1\n",422 " \n",423 " print(\"📚 Installing dependencies (this takes ~10 seconds)...\")\n",424 " !pip install -q fastapi uvicorn requests fastmcp\n",425 " \n",426 " import sys\n",427 " sys.path.insert(0, './src')\n",428 " print(\"\\n✅ Setup complete! Everything is ready to go! 🎉\")\n",429 "else:\n",430 " import sys\n",431 " from pathlib import Path\n",432 " sys.path.insert(0, str(Path.cwd().parent))\n",433 " print(\"✅ Using local OpenEnv installation\")\n",434 "\n",435 "print(\"\\n🚀 Ready to explore OpenEnv and build amazing things!\")\n",436 "print(\"💡 Tip: Run cells top-to-bottom for the best experience.\\n\")"437 ]438 },439 {440 "cell_type": "markdown",441 "metadata": {},442 "source": [443 "---\n",444 "\n",445 "<a id=\"part-4\"></a>\n",446 "# Part 4: The OpenEnv Pattern 🏗️\n",447 "\n",448 "<div style=\"background-color: #f0f7ff; padding: 20px; border-radius: 10px; margin: 20px 0;\">\n",449 "\n",450 "## Every OpenEnv Environment Has 3 Components:\n",451 "\n",452 "```\n",453 "envs/your_env/\n",454 "├── 📝 models.py ← Type-safe contracts\n",455 "│ (Action, Observation, State)\n",456 "│\n",457 "├── 📱 client.py ← What YOU import\n",458 "│ (HTTPEnvClient implementation)\n",459 "│\n",460 "└── 🖥️ server/\n",461 " ├── environment.py ← Game/simulation logic\n",462 " ├── app.py ← FastAPI server\n",463 " └── Dockerfile ← Container definition\n",464 "```\n",465 "\n",466 "</div>\n",467 "\n",468 "Let's explore the actual OpenEnv code to see how this works:"469 ]470 },471 {472 "cell_type": "code",473 "execution_count": 3,474 "metadata": {},475 "outputs": [476 {477 "name": "stdout",478 "output_type": "stream",479 "text": [480 "======================================================================\n",481 " 🧩 OPENENV CORE ABSTRACTIONS\n",482 "======================================================================\n",483 "\n",484 "🖥️ SERVER SIDE (runs in Docker):\n",485 "\n",486 " class Environment(ABC):\n",487 " '''Base class for all environment implementations'''\n",488 "\n",489 " @abstractmethod\n",490 " def reset(self) -> Observation:\n",491 " '''Start new episode'''\n",492 "\n",493 " @abstractmethod\n",494 " def step(self, action: Action) -> Observation:\n",495 " '''Execute action, return observation'''\n",496 "\n",497 " @property\n",498 " def state(self) -> State:\n",499 " '''Get episode metadata'''\n",500 "\n",501 "📱 CLIENT SIDE (your training code):\n",502 "\n",503 " class EnvClient(ABC):\n",504 " '''Base class for HTTP clients'''\n",505 "\n",506 " def reset(self) -> StepResult:\n",507 " # HTTP POST /reset\n",508 "\n",509 " def step(self, action) -> StepResult:\n",510 " # HTTP POST /step\n",511 "\n",512 " def state(self) -> State:\n",513 " # HTTP GET /state\n",514 "\n",515 "======================================================================\n",516 "\n",517 "✨ Same interface on both sides - communication via HTTP!\n",518 "🎯 You focus on RL, OpenEnv handles the infrastructure.\n",519 "\n"520 ]521 }522 ],523 "source": [524 "# Import OpenEnv's core abstractions\n",525 "from openenv.core.env_server import Environment, Action, Observation, State\n",526 "from openenv.core.env_client import EnvClient\n",527 "\n",528 "print(\"=\"*70)\n",529 "print(\" 🧩 OPENENV CORE ABSTRACTIONS\")\n",530 "print(\"=\"*70)\n",531 "\n",532 "print(\"\"\"\n",533 "🖥️ SERVER SIDE (runs in Docker):\n",534 "\n",535 " class Environment(ABC):\n",536 " '''Base class for all environment implementations'''\n",537 " \n",538 " @abstractmethod\n",539 " def reset(self) -> Observation:\n",540 " '''Start new episode'''\n",541 " \n",542 " @abstractmethod\n",543 " def step(self, action: Action) -> Observation:\n",544 " '''Execute action, return observation'''\n",545 " \n",546 " @property\n",547 " def state(self) -> State:\n",548 " '''Get episode metadata'''\n",549 "\n",550 "📱 CLIENT SIDE (your training code):\n",551 "\n",552 " class EnvClient(ABC):\n",553 " '''Base class for HTTP clients'''\n",554 " \n",555 " def reset(self) -> StepResult:\n",556 " # HTTP POST /reset\n",557 " \n",558 " def step(self, action) -> StepResult:\n",559 " # HTTP POST /step\n",560 " \n",561 " def state(self) -> State:\n",562 " # HTTP GET /state\n",563 "\"\"\")\n",564 "\n",565 "print(\"=\"*70)\n",566 "print(\"\\n✨ Same interface on both sides - communication via HTTP!\")\n",567 "print(\"🎯 You focus on RL, OpenEnv handles the infrastructure.\\n\")"568 ]569 },570 {571 "cell_type": "markdown",572 "metadata": {},573 "source": [574 "---\n",575 "\n",576 "# Part 5: Example Integration - OpenSpiel 🎮\n",577 "\n",578 "<div style=\"background-color: #fff3e0; padding: 20px; border-radius: 10px; margin: 20px 0;\">\n",579 "\n",580 "## What is OpenSpiel?\n",581 "\n",582 "**OpenSpiel** is a library from DeepMind with **70+ game environments** for RL research.\n",583 "\n",584 "## OpenEnv's Integration\n",585 "\n",586 "We've wrapped **6 OpenSpiel games** following the OpenEnv pattern:\n",587 "\n",588 "<table>\n",589 "<tr>\n",590 "<td width=\"50%\">\n",591 "\n",592 "**🎯 Single-Player**\n",593 "1. **Catch** - Catch falling ball\n",594 "2. **Cliff Walking** - Navigate grid\n",595 "3. **2048** - Tile puzzle\n",596 "4. **Blackjack** - Card game\n",597 "\n",598 "</td>\n",599 "<td width=\"50%\">\n",600 "\n",601 "**👥 Multi-Player**\n",602 "5. **Tic-Tac-Toe** - Classic 3×3\n",603 "6. **Kuhn Poker** - Imperfect info poker\n",604 "\n",605 "</td>\n",606 "</tr>\n",607 "</table>\n",608 "\n",609 "This shows how OpenEnv can wrap **any** existing RL library!\n",610 "\n",611 "</div>"612 ]613 },614 {615 "cell_type": "code",616 "execution_count": 4,617 "metadata": {},618 "outputs": [619 {620 "name": "stdout",621 "output_type": "stream",622 "text": [623 "======================================================================\n",624 " 🔌 HOW OPENENV WRAPS OPENSPIEL\n",625 "======================================================================\n",626 "\n",627 "class OpenSpielEnv(HTTPEnvClient[OpenSpielAction, OpenSpielObservation]):\n",628 "\n",629 " def _step_payload(self, action: OpenSpielAction) -> dict:\n",630 " '''Convert typed action to JSON for HTTP'''\n",631 " return {\n",632 " \"action_id\": action.action_id,\n",633 " \"game_name\": action.game_name,\n",634 " }\n",635 "\n",636 " def _parse_result(self, payload: dict) -> StepResult:\n",637 " '''Parse HTTP JSON response into typed observation'''\n",638 " return StepResult(\n",639 " observation=OpenSpielObservation(...),\n",640 " reward=payload['reward'],\n",641 " done=payload['done']\n",642 " )\n",643 "\n",644 "\n",645 "──────────────────────────────────────────────────────────────────────\n",646 "\n",647 "✨ Usage (works for ALL OpenEnv environments):\n",648 "\n",649 " env = OpenSpielEnv(base_url=\"http://localhost:8000\")\n",650 "\n",651 " result = env.reset()\n",652 " # Returns StepResult[OpenSpielObservation] - Type safe!\n",653 "\n",654 " result = env.step(OpenSpielAction(action_id=2, game_name=\"catch\"))\n",655 " # Type checker knows this is valid!\n",656 "\n",657 " state = env.state()\n",658 " # Returns OpenSpielState\n",659 "\n",660 "──────────────────────────────────────────────────────────────────────\n",661 "\n",662 "🎯 This pattern works for ANY environment you want to wrap!\n",663 "\n"664 ]665 }666 ],667 "source": [668 "from envs.openspiel_env.client import OpenSpielEnv\n",669 "\n",670 "print(\"=\"*70)\n",671 "print(\" 🔌 HOW OPENENV WRAPS OPENSPIEL\")\n",672 "print(\"=\"*70)\n",673 "\n",674 "print(\"\"\"\n",675 "class OpenSpielEnv(HTTPEnvClient[OpenSpielAction, OpenSpielObservation]):\n",676 " \n",677 " def _step_payload(self, action: OpenSpielAction) -> dict:\n",678 " '''Convert typed action to JSON for HTTP'''\n",679 " return {\n",680 " \"action_id\": action.action_id,\n",681 " \"game_name\": action.game_name,\n",682 " }\n",683 " \n",684 " def _parse_result(self, payload: dict) -> StepResult:\n",685 " '''Parse HTTP JSON response into typed observation'''\n",686 " return StepResult(\n",687 " observation=OpenSpielObservation(...),\n",688 " reward=payload['reward'],\n",689 " done=payload['done']\n",690 " )\n",691 "\n",692 "\"\"\")\n",693 "\n",694 "print(\"─\" * 70)\n",695 "print(\"\\n✨ Usage (works for ALL OpenEnv environments):\")\n",696 "print(\"\"\"\n",697 " env = OpenSpielEnv(base_url=\"http://localhost:8000\")\n",698 " \n",699 " result = env.reset()\n",700 " # Returns StepResult[OpenSpielObservation] - Type safe!\n",701 " \n",702 " result = env.step(OpenSpielAction(action_id=2, game_name=\"catch\"))\n",703 " # Type checker knows this is valid!\n",704 " \n",705 " state = env.state()\n",706 " # Returns OpenSpielState\n",707 "\"\"\")\n",708 "\n",709 "print(\"─\" * 70)\n",710 "print(\"\\n🎯 This pattern works for ANY environment you want to wrap!\\n\")"711 ]712 },713 {714 "cell_type": "code",715 "execution_count": 5,716 "metadata": {},717 "outputs": [718 {719 "name": "stdout",720 "output_type": "stream",721 "text": [722 "======================================================================\n",723 " 🎮 OPENSPIEL INTEGRATION - TYPE-SAFE MODELS\n",724 "======================================================================\n",725 "\n",726 "📤 OpenSpielAction (what you send):\n",727 " ────────────────────────────────────────────────────────────────\n",728 " • metadata : typing.Dict[str, typing.Any]\n",729 " • action_id : <class 'int'>\n",730 " • game_name : <class 'str'>\n",731 " • game_params : typing.Dict[str, typing.Any]\n",732 "\n",733 "📥 OpenSpielObservation (what you receive):\n",734 " ────────────────────────────────────────────────────────────────\n",735 " • done : <class 'bool'>\n",736 " • reward : bool | int | float | None\n",737 " • metadata : typing.Dict[str, typing.Any]\n",738 " • info_state : typing.List[float]\n",739 " • legal_actions : typing.List[int]\n",740 " • game_phase : <class 'str'>\n",741 " • current_player_id : <class 'int'>\n",742 " • opponent_last_action : typing.Optional[int]\n",743 "\n",744 "📊 OpenSpielState (episode metadata):\n",745 " ────────────────────────────────────────────────────────────────\n",746 " • episode_id : typing.Optional[str]\n",747 " • step_count : <class 'int'>\n",748 " • game_name : <class 'str'>\n",749 " • agent_player : <class 'int'>\n",750 " • opponent_policy : <class 'str'>\n",751 " • game_params : typing.Dict[str, typing.Any]\n",752 " • num_players : <class 'int'>\n",753 "\n",754 "======================================================================\n",755 "\n",756 "💡 Type safety means:\n",757 " ✅ Your IDE autocompletes these fields\n",758 " ✅ Typos are caught before running\n",759 " ✅ Refactoring is safe\n",760 " ✅ Self-documenting code\n",761 "\n"762 ]763 }764 ],765 "source": [766 "from envs.openspiel_env.models import (\n",767 " OpenSpielAction,\n",768 " OpenSpielObservation,\n",769 " OpenSpielState\n",770 ")\n",771 "\n",772 "print(\"=\" * 70)\n",773 "print(\" 🎮 OPENSPIEL INTEGRATION - TYPE-SAFE MODELS\")\n",774 "print(\"=\" * 70)\n",775 "\n",776 "print(\"\\n📤 OpenSpielAction (what you send):\")\n",777 "print(\" \" + \"─\" * 64)\n",778 "for name, field in OpenSpielAction.model_fields.items():\n",779 " print(f\" • {name:20s} : {field.annotation}\")\n",780 "\n",781 "print(\"\\n📥 OpenSpielObservation (what you receive):\")\n",782 "print(\" \" + \"─\" * 64)\n",783 "for name, field in OpenSpielObservation.model_fields.items():\n",784 " print(f\" • {name:20s} : {field.annotation}\")\n",785 "\n",786 "print(\"\\n📊 OpenSpielState (episode metadata):\")\n",787 "print(\" \" + \"─\" * 64)\n",788 "for name, field in OpenSpielState.model_fields.items():\n",789 " print(f\" • {name:20s} : {field.annotation}\")\n",790 "\n",791 "print(\"\\n\" + \"=\" * 70)\n",792 "print(\"\\n💡 Type safety means:\")\n",793 "print(\" ✅ Your IDE autocompletes these fields\")\n",794 "print(\" ✅ Typos are caught before running\")\n",795 "print(\" ✅ Refactoring is safe\")\n",796 "print(\" ✅ Self-documenting code\\n\")\n"797 ]798 },799 {800 "cell_type": "markdown",801 "metadata": {},802 "source": [803 "### How the Client Works\n",804 "\n",805 "<div style=\"background-color: rgba(128, 128, 128, 0.1); padding: 20px; border-radius: 10px; margin: 20px 0; border: 1px solid rgba(128, 128, 128, 0.2);\">\n",806 "\n",807 "The client **inherits from HTTPEnvClient** and implements 3 methods:\n",808 "\n",809 "1. `_step_payload()` - Convert action → JSON\n",810 "2. `_parse_result()` - Parse JSON → typed observation \n",811 "3. `_parse_state()` - Parse JSON → state\n",812 "\n",813 "That's it! The base class handles all HTTP communication.\n",814 "\n",815 "</div>"816 ]817 },818 {819 "cell_type": "markdown",820 "metadata": {},821 "source": [822 "---\n",823 "\n",824 "<div style=\"text-align: center; background: linear-gradient(135deg, #667eea 0%, #764ba2 100%); color: white; padding: 30px; border-radius: 15px; margin: 30px 0;\">\n",825 "\n",826 "# 🎮 Part 6: Using Real OpenSpiel\n",827 "\n",828 "### Now let's USE a production environment!\n",829 "\n",830 "We'll play **Catch** using OpenEnv's **OpenSpiel integration** 🎯<br>\n",831 "This is a REAL environment running in production at companies!\n",832 "\n",833 "<br>\n",834 "\n",835 "**Get ready for:**\n",836 "- 🔌 Using existing environments (not building)\n",837 "- 🤖 Testing policies against real games\n",838 "- 📊 Live gameplay visualization\n",839 "- 🎯 Production-ready patterns\n",840 "\n",841 "</div>"842 ]843 },844 {845 "cell_type": "markdown",846 "metadata": {},847 "source": [848 "## The Game: Catch 🔴🏓\n",849 "\n",850 "<table>\n",851 "<tr>\n",852 "<td width=\"40%\" style=\"text-align: center;\">\n",853 "\n",854 "```\n",855 "⬜ ⬜ 🔴 ⬜ ⬜\n",856 "⬜ ⬜ ⬜ ⬜ ⬜\n",857 "⬜ ⬜ ⬜ ⬜ ⬜ Ball\n",858 "⬜ ⬜ ⬜ ⬜ ⬜\n",859 "⬜ ⬜ ⬜ ⬜ ⬜ falls\n",860 "⬜ ⬜ ⬜ ⬜ ⬜\n",861 "⬜ ⬜ ⬜ ⬜ ⬜ down\n",862 "⬜ ⬜ ⬜ ⬜ ⬜\n",863 "⬜ ⬜ ⬜ ⬜ ⬜\n",864 "⬜ ⬜ 🏓 ⬜ ⬜\n",865 " Paddle\n",866 "```\n",867 "\n",868 "</td>\n",869 "<td width=\"60%\">\n",870 "\n",871 "**Rules:**\n",872 "- 10×5 grid\n",873 "- Ball falls from random column\n",874 "- Move paddle left/right to catch it\n",875 "\n",876 "**Actions:**\n",877 "- `0` = Move LEFT ⬅️\n",878 "- `1` = STAY 🛑\n",879 "- `2` = Move RIGHT ➡️\n",880 "\n",881 "**Reward:**\n",882 "- `+1` if caught 🎉\n",883 "- `0` if missed 😢\n",884 "\n",885 "</td>\n",886 "</tr>\n",887 "</table>\n",888 "\n",889 "<div style=\"background-color: rgba(128, 128, 128, 0.1); padding: 20px; border-radius: 10px; margin: 20px 0; border: 1px solid rgba(128, 128, 128, 0.2);\">\n",890 "\n",891 "**🎯 Why Catch?**\n",892 "- Simple rules (easy to understand)\n",893 "- Fast episodes (~5 steps)\n",894 "- Clear success/failure\n",895 "- Part of OpenSpiel's 70+ games!\n",896 "\n",897 "**💡 The Big Idea:**\n",898 "Instead of building this from scratch, we'll USE OpenEnv's existing OpenSpiel integration. Same interface, but production-ready!\n",899 "\n",900 "</div>"901 ]902 },903 {904 "cell_type": "code",905 "execution_count": 6,906 "metadata": {},907 "outputs": [908 {909 "name": "stdout",910 "output_type": "stream",911 "text": [912 "🎮 ================================================================ 🎮\n",913 " ✅ Importing Real OpenSpiel Environment!\n",914 "🎮 ================================================================ 🎮\n",915 "\n",916 "📦 What we just imported:\n",917 " • OpenSpielEnv - HTTP client for OpenSpiel games\n",918 " • OpenSpielAction - Type-safe actions\n",919 " • OpenSpielObservation - Type-safe observations\n",920 " • OpenSpielState - Episode metadata\n",921 "\n",922 "📋 OpenSpielObservation fields:\n",923 " ────────────────────────────────────────────────────────────\n",924 " • done : <class 'bool'>\n",925 " • reward : bool | int | float | None\n",926 " • metadata : typing.Dict[str, typing.Any]\n",927 " • info_state : typing.List[float]\n",928 " • legal_actions : typing.List[int]\n",929 " • game_phase : <class 'str'>\n",930 " • current_player_id : <class 'int'>\n",931 " • opponent_last_action : typing.Optional[int]\n",932 "\n",933 "======================================================================\n",934 "\n",935 "💡 This is REAL OpenEnv code - used in production!\n",936 " • Wraps 6 OpenSpiel games (Catch, Tic-Tac-Toe, Poker, etc.)\n",937 " • Type-safe actions and observations\n",938 " • Works via HTTP (we'll see that next!)\n",939 "\n"940 ]941 }942 ],943 "source": [944 "from envs.openspiel_env import OpenSpielEnv\n",945 "from envs.openspiel_env.models import (\n",946 " OpenSpielAction,\n",947 " OpenSpielObservation,\n",948 " OpenSpielState,\n",949 ")\n",950 "\n",951 "print(\"🎮 \" + \"=\" * 64 + \" 🎮\")\n",952 "print(\" ✅ Importing Real OpenSpiel Environment!\")\n",953 "print(\"🎮 \" + \"=\" * 64 + \" 🎮\\n\")\n",954 "\n",955 "print(\"📦 What we just imported:\")\n",956 "print(\" • OpenSpielEnv - HTTP client for OpenSpiel games\")\n",957 "print(\" • OpenSpielAction - Type-safe actions\")\n",958 "print(\" • OpenSpielObservation - Type-safe observations\")\n",959 "print(\" • OpenSpielState - Episode metadata\\n\")\n",960 "\n",961 "print(\"📋 OpenSpielObservation fields:\")\n",962 "print(\" \" + \"─\" * 60)\n",963 "for name, field in OpenSpielObservation.model_fields.items():\n",964 " print(f\" • {name:25s} : {field.annotation}\")\n",965 "\n",966 "print(\"\\n\" + \"=\" * 70)\n",967 "print(\"\\n💡 This is REAL OpenEnv code - used in production!\")\n",968 "print(\" • Wraps 6 OpenSpiel games (Catch, Tic-Tac-Toe, Poker, etc.)\")\n",969 "print(\" • Type-safe actions and observations\")\n",970 "print(\" • Works via HTTP (we'll see that next!)\\n\")"971 ]972 },973 {974 "cell_type": "code",975 "execution_count": 7,976 "metadata": {},977 "outputs": [978 {979 "name": "stdout",980 "output_type": "stream",981 "text": [982 "🚀 ================================================================ 🚀\n",983 " Starting OpenSpiel Server (Catch Game)\n",984 "🚀 ================================================================ 🚀\n",985 "\n",986 "✅ OpenSpiel is installed!\n",987 "\n",988 "⚡ Starting FastAPI server for OpenSpiel Catch...\n",989 " (This uses REAL OpenEnv + OpenSpiel integration)\n",990 "\n",991 "⏳ Waiting for server to start...\n",992 "\n",993 "✅ OpenSpiel server is running!\n",994 "🌐 Server URL: http://localhost:8000\n",995 "📍 Endpoints available:\n",996 " • POST /reset\n",997 " • POST /step\n",998 " • GET /state\n",999 "\n",1000 "🎯 This is REAL OpenEnv + OpenSpiel in action!\n",1001 " • Running actual OpenSpiel Catch game\n",1002 " • Exposed via FastAPI HTTP server\n",1003 " • Using OpenEnv's standard interface\n",1004 "\n"1005 ]1006 }1007 ],1008 "source": [1009 "import subprocess\n",1010 "import time\n",1011 "import sys\n",1012 "import os\n",1013 "\n",1014 "print(\"🚀 \" + \"=\"*64 + \" 🚀\")\n",1015 "print(\" Starting OpenSpiel Server (Catch Game)\")\n",1016 "print(\"🚀 \" + \"=\"*64 + \" 🚀\\n\")\n",1017 "\n",1018 "# Check if open_spiel is installed\n",1019 "try:\n",1020 " import pyspiel\n",1021 " print(\"✅ OpenSpiel is installed!\\n\")\n",1022 "except ImportError:\n",1023 " print(\"⚠️ OpenSpiel not found. Installing...\")\n",1024 " import subprocess\n",1025 " subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"open_spiel\"])\n",1026 " print(\"✅ OpenSpiel installed!\\n\")\n",1027 "\n",1028 "# Start the OpenSpiel server in background\n",1029 "print(\"⚡ Starting FastAPI server for OpenSpiel Catch...\")\n",1030 "print(\" (This uses REAL OpenEnv + OpenSpiel integration)\\n\")\n",1031 "\n",1032 "# Determine the correct path\n",1033 "if IN_COLAB:\n",1034 " work_dir = \"/content/OpenEnv\"\n",1035 "else:\n",1036 " from pathlib import Path\n",1037 " work_dir = str(Path.cwd().parent.absolute())\n",1038 "\n",1039 "server_process = subprocess.Popen(\n",1040 " [sys.executable, \"-m\", \"uvicorn\",\n",1041 " \"envs.openspiel_env.server.app:app\",\n",1042 " \"--host\", \"0.0.0.0\",\n",1043 " \"--port\", \"8000\"],\n",1044 " env={**os.environ,\n",1045 " \"PYTHONPATH\": f\"{work_dir}/src\",\n",1046 " \"OPENSPIEL_GAME\": \"catch\",\n",1047 " \"OPENSPIEL_AGENT_PLAYER\": \"0\",\n",1048 " \"OPENSPIEL_OPPONENT_POLICY\": \"random\"},\n",1049 " stdout=subprocess.DEVNULL,\n",1050 " stderr=subprocess.DEVNULL,\n",1051 " text=True,\n",1052 " cwd=work_dir\n",1053 ")\n",1054 "\n",1055 "# Wait for server to start\n",1056 "print(\"⏳ Waiting for server to start...\")\n",1057 "time.sleep(5)\n",1058 "\n",1059 "# Check if server is running\n",1060 "import requests\n",1061 "try:\n",1062 " response = requests.get('http://localhost:8000/health', timeout=2)\n",1063 " print(\"\\n✅ OpenSpiel server is running!\")\n",1064 " print(\"🌐 Server URL: http://localhost:8000\")\n",1065 " print(\"📍 Endpoints available:\")\n",1066 " print(\" • POST /reset\")\n",1067 " print(\" • POST /step\")\n",1068 " print(\" • GET /state\")\n",1069 " print(\"\\n🎯 This is REAL OpenEnv + OpenSpiel in action!\")\n",1070 " print(\" • Running actual OpenSpiel Catch game\")\n",1071 " print(\" • Exposed via FastAPI HTTP server\")\n",1072 " print(\" • Using OpenEnv's standard interface\\n\")\n",1073 "except Exception as e:\n",1074 " print(f\"\\n❌ Server failed to start: {e}\")\n",1075 " print(\"\\n📋 Checking error output...\")\n",1076 " server_process.poll()\n",1077 " if server_process.stderr:\n",1078 " stderr = server_process.stderr.read()\n",1079 " if stderr:\n",1080 " print(stderr)\n",1081 " print(\"\\n💡 Make sure open_spiel is installed:\")\n",1082 " print(\" pip install open_spiel\")\n",1083 " raise"1084 ]1085 },1086 {1087 "cell_type": "code",1088 "execution_count": 8,1089 "metadata": {},1090 "outputs": [1091 {1092 "name": "stdout",1093 "output_type": "stream",1094 "text": [1095 "📱 ================================================================ 📱\n",1096 " Connecting to OpenSpiel Server via HTTP\n",1097 "📱 ================================================================ 📱\n",1098 "\n",1099 "✅ Client created!\n",1100 "\n",1101 "💡 What just happened:\n",1102 " • OpenSpielEnv is an HTTPEnvClient subclass\n",1103 " • It knows how to talk to OpenSpiel servers\n",1104 " • All communication is type-safe and over HTTP\n",1105 " • Same client works for ALL OpenSpiel games!\n",1106 "\n"1107 ]1108 }1109 ],1110 "source": [1111 "print(\"📱 \" + \"=\"*64 + \" 📱\")\n",1112 "print(\" Connecting to OpenSpiel Server via HTTP\")\n",1113 "print(\"📱 \" + \"=\"*64 + \" 📱\\n\")\n",1114 "\n",1115 "# Create HTTP client for OpenSpiel\n",1116 "client = OpenSpielEnv(base_url=\"http://localhost:8000\")\n",1117 "\n",1118 "print(\"✅ Client created!\")\n",1119 "print(\"\\n💡 What just happened:\")\n",1120 "print(\" • OpenSpielEnv is an HTTPEnvClient subclass\")\n",1121 "print(\" • It knows how to talk to OpenSpiel servers\")\n",1122 "print(\" • All communication is type-safe and over HTTP\")\n",1123 "print(\" • Same client works for ALL OpenSpiel games!\\n\")"1124 ]1125 },1126 {1127 "cell_type": "code",1128 "execution_count": 9,1129 "metadata": {},1130 "outputs": [1131 {1132 "name": "stdout",1133 "output_type": "stream",1134 "text": [1135 "🎮 ================================================================ 🎮\n",1136 " Testing Connection - Playing One Step\n",1137 "🎮 ================================================================ 🎮\n",1138 "\n",1139 "📤 Calling client.reset()...\n",1140 " Under the hood: HTTP POST to http://localhost:8000/reset\n",1141 "\n",1142 "📥 Received OpenSpielObservation:\n",1143 " • info_state: [0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]... (first 10 values)\n",1144 " • number of info_state: 50\n",1145 " • legal_actions: [0, 1, 2]\n",1146 " • game_phase: initial\n",1147 " • done: False\n",1148 "\n",1149 "📤 Calling client.step(OpenSpielAction(action_id=1, game_name='catch'))...\n",1150 " Under the hood: HTTP POST to http://localhost:8000/step\n",1151 "\n",1152 "📥 Received response:\n",1153 " • Reward: 0.0\n",1154 " • Done: False\n",1155 " • legal_actions: [0, 1, 2]\n",1156 "\n",1157 "📊 Episode state:\n",1158 " • episode_id: aec2e599-5633-45d6-a7a3-30f46e8796ab\n",1159 " • step_count: 1\n",1160 " • game_name: catch\n",1161 "\n",1162 "======================================================================\n",1163 "\n",1164 "🎉 IT WORKS! We're using REAL OpenSpiel via HTTP!\n",1165 " ✅ Type-safe communication\n",1166 " ✅ Same interface as any OpenEnv environment\n",1167 " ✅ Production-ready architecture\n",1168 "\n"1169 ]1170 }1171 ],1172 "source": [1173 "print(\"🎮 \" + \"=\"*64 + \" 🎮\")\n",1174 "print(\" Testing Connection - Playing One Step\")\n",1175 "print(\"🎮 \" + \"=\"*64 + \" 🎮\\n\")\n",1176 "\n",1177 "# Reset the environment (HTTP POST /reset)\n",1178 "print(\"📤 Calling client.reset()...\")\n",1179 "print(\" Under the hood: HTTP POST to http://localhost:8000/reset\\n\")\n",1180 "\n",1181 "result = client.reset()\n",1182 "\n",1183 "print(\"📥 Received OpenSpielObservation:\")\n",1184 "print(f\" • info_state: {result.observation.info_state[:10]}... (first 10 values)\")\n",1185 "print(f\" • number of info_state: {len(result.observation.info_state)}\")\n",1186 "print(f\" • legal_actions: {result.observation.legal_actions}\")\n",1187 "print(f\" • game_phase: {result.observation.game_phase}\")\n",1188 "print(f\" • done: {result.done}\")\n",1189 "\n",1190 "# Take an action (HTTP POST /step)\n",1191 "print(\"\\n📤 Calling client.step(OpenSpielAction(action_id=1, game_name=\\'catch\\'))...\")\n",1192 "print(\" Under the hood: HTTP POST to http://localhost:8000/step\\n\")\n",1193 "\n",1194 "action = OpenSpielAction(action_id=1, game_name=\"catch\") # STAY\n",1195 "result = client.step(action)\n",1196 "\n",1197 "print(\"📥 Received response:\")\n",1198 "print(f\" • Reward: {result.reward}\")\n",1199 "print(f\" • Done: {result.done}\")\n",1200 "print(f\" • legal_actions: {result.observation.legal_actions}\")\n",