Ab-Romia/grid-world-rl-game

🎮 Grid World RL Game

An interactive reinforcement learning game where you can watch an AI agent learn to navigate a grid world in real-time! Built with Gradio and deployable on Hugging Face Spaces.

🌟 Features

• —Interactive Web Interface: No coding required - just adjust parameters and watch the agent learn
• —Real-time Visualization: See the agent's policy, value function, and decision-making process
• —Multiple Algorithms: Compare Value Iteration and Policy Iteration
• —Customizable Environment: Adjust grid size, rewards, and discount factors
• —Live Simulations: Watch trained agents navigate the environment step-by-step
• —Educational: Perfect for learning RL concepts visually

🚀 Quick Start

Local Installation

# Clone the repository
git clone https://github.com/Ab-Romia/Reinforcement_Learning.git
cd Reinforcement_Learning

# Install dependencies
pip install -r requirements.txt

# Run the app
python app.py

The app will open in your browser at http://localhost:7860

Hugging Face Spaces

This app is ready to deploy on Hugging Face Spaces:

1. 1.Create a new Space on Hugging Face
2. 2.Choose "Gradio" as the SDK
3. 3.Upload all files from this repository
4. 4.The app will automatically deploy!

🎯 How It Works

The Environment

The agent navigates a grid world with:

• —Terminal States (★): Episodes end here with big rewards or penalties
• —Top-right corner: +100 reward
• —Bottom-left corner: -50 penalty
• —Reward Cells (◆): Intermediate rewards the agent can collect
• —Regular Cells: Small negative reward (-1) to encourage efficiency
• —Stochastic Movement: 70% chance of intended direction, 15% each perpendicular

The Learning Process

The agent uses Reinforcement Learning algorithms to find the optimal policy:

1. 1.Value Iteration:
2. 2.Iteratively computes the value of each state
3. 3.Converges to optimal policy
4. 4.Uses the Bellman optimality equation

1. 1.Policy Iteration:
2. 2.Alternates between policy evaluation and improvement
3. 3.Often converges faster than value iteration
4. 4.Guaranteed to find optimal policy

Key Concepts

• —Policy: The agent's strategy - which action to take in each state
• —Value Function: Expected total reward from each state
• —Discount Factor (γ): How much the agent values future vs immediate rewards
• —Stochastic Transitions: Movement isn't always predictable, like in real life!

🎮 Using the Interface

Training Tab

1. 1.Configure Parameters:
2. 2.Grid Size: 3x3 to 8x8
3. 3.Center Reward: -50 to +50
4. 4.Discount Factor: 0.8 to 0.99
5. 5.Algorithm: Value or Policy Iteration

1. 1.Click "Train Agent" to run the learning algorithm

1. 1.View Results:
2. 2.Learned policy with arrows showing optimal actions
3. 3.Value function as background colors
4. 4.Convergence statistics

Simulation Tab

Watch a trained agent navigate the environment:

• —Stochastic movement (more realistic)
• —Step-by-step visualization
• —Total reward tracking

🔬 Experiment Ideas

Try these configurations to see different behaviors:

1. 1.Risk Taker vs Risk Avoider:
2. 2.High γ (0.99): Values future rewards, takes longer paths
3. 3.Low γ (0.85): Prefers quick rewards, takes shortcuts

1. 1.Reward Attraction:
2. 2.Positive center reward (+30): Agent seeks it out
3. 3.Negative center reward (-30): Agent avoids it

1. 1.Grid Complexity:
2. 2.Small grid (3x3): Simple, fast learning
3. 3.Large grid (8x8): Complex, more interesting policies

1. 1.Penalty Sensitivity:
2. 2.Adjust bottom-left penalty and see how the agent's path changes

📁 Project Structure

Reinforcement_Learning/
├── app.py              # Gradio web interface
├── game.py             # Grid World game logic
├── flame.py            # Original MDP implementation
├── requirements.txt    # Python dependencies
└── README.md          # This file

🧠 Technical Details

Algorithms

Value Iteration:

V(s) ← max_a Σ P(s'|s,a)[R(s) + γV(s')]

Policy Iteration:

1. Policy Evaluation: Compute V^π(s)
2. Policy Improvement: π'(s) ← argmax_a Q(s,a)
3. Repeat until convergence

Transition Model

For action a from state s:

• —70% probability: intended direction
• —15% probability: perpendicular left
• —15% probability: perpendicular right
• —If move would exit grid: stay in current state

Rewards

• —Terminal positive: +100
• —Terminal negative: -50
• —Center cell: configurable
• —Step cost: -1 (encourages efficiency)

🎨 Visualization Features

• —Color-coded values: Warmer colors = higher values
• —Policy arrows: Clear direction indicators
• —Special state markers:
• —★ Terminal states
• —◆ Reward cells
• —🤖 Agent position
• —Real-time updates: See learning progress

🔧 Advanced Usage

Programmatic Access

from game import GridWorldGame

# Create custom environment
game = GridWorldGame(r_value=20, gamma=0.95, grid_size=6)

# Train agent
values, iterations = game.value_iteration()
policy = game.extract_policy(values)

# Run episode
trajectory, reward = game.run_episode(policy)
print(f"Total reward: {reward}")

# Visualize
fig = game.visualize_grid(values, policy)

Custom Environments

Modify game.py to create your own environments:

• —Change terminal state positions
• —Add more reward cells
• —Adjust transition probabilities
• —Implement obstacles

📚 Learning Resources

Great for understanding:

• —Markov Decision Processes (MDPs)
• —Dynamic Programming in RL
• —Value and Policy Iteration
• —Stochastic environments
• —Discount factors and their effects

🤝 Contributing

This is an educational project. Feel free to:

• —Add new RL algorithms (Q-learning, SARSA, etc.)
• —Improve visualizations
• —Add new environment features
• —Create tutorials

📄 License

MIT License - feel free to use for learning and teaching!

🙏 Acknowledgments

Built with:

• —Gradio - Easy ML web interfaces
• —NumPy - Numerical computing
• —Matplotlib - Visualization

Inspired by classic RL textbooks:

• —Sutton & Barto - "Reinforcement Learning: An Introduction"
• —Russell & Norvig - "Artificial Intelligence: A Modern Approach"

📧 Contact

Questions or suggestions? Open an issue or reach out!

Have fun learning Reinforcement Learning! 🚀

Try different configurations, observe how the agent learns, and build intuition for RL concepts. The best way to learn is by experimenting!