ruddyscent/gymnasium-playbook-frozenlake-q-learning

FrozenLake 4x4: tabular Q-learning

Three trained Q-tables (training seeds 0, 1, and 2) from gymnasium-playbook. These are NumPy tables, not neural network weights.

Environment and training

FrozenLake-v1, map_name="4x4", is_slippery=False, and a 100-step TimeLimit. Observations are integer states (0–15); actions use Gymnasium's standard ordering: LEFT, DOWN, RIGHT, UP. No custom preprocessing or wrappers are used beyond Gymnasium's standard environment wrappers and TimeLimit.

Each table was trained for 10,000 episodes with learning rate 0.1, discount 0.99, and epsilon decaying from 1.0 by a factor of 0.999 per episode to a minimum of 0.05. Per-seed config.json records the complete configuration and original software versions: Python 3.13.15, Gymnasium 1.3.0, NumPy 2.5.3, macOS ARM64.

Evaluation

Greedy evaluation uses int(np.argmax(q_table[state])), including the first maximum as the deterministic tie rule. Each policy was evaluated for 1,000 episodes with evaluation seed 10000 + training_seed and a 100-step limit.

All six evaluations were rerun from the saved tables before packaging, and matched the original reports exactly. No evaluation episode was truncated. These results apply only to the deterministic 4x4 map; they do not establish performance on slippery dynamics, other maps, or larger environments.

Files

Each seed-N/ contains q_table.npy (16 by 4), config.json, training.csv, and evaluation.json. Top-level summaries compare all seeds. provenance.json records artifact SHA-256 hashes and the code revision used for verification. The original training commit was not recorded; the verification commit must not be treated as proven training provenance.

The tables support greedy inference. Exact training resumption is not supported: RNG state and a complete resumable training state were not saved.

Loading and evaluation

Download this repository at a specific immutable Hub commit revision and keep that revision with your experiment records. In the source checkout at commit 4a142954fb52ef8e052257407d85a1d7884d9af3, use the pinned uv setup from its README and run:

uv sync --locked
uv run --locked python -m environments.toy_text.frozen_lake evaluate --q-table /path/to/download/seed-0/q_table.npy --episodes 1000 --seed 10000 --max-episode-steps 100

Replace /path/to/download with the local download directory. For seeds 1 and 2, use their respective file and evaluation seeds 10001 and 10002. Load tables without pickle when using NumPy directly:

import numpy as np
q_table = np.load("seed-0/q_table.npy", allow_pickle=False)
action = int(np.argmax(q_table[state]))

Here state is the current integer observation from the configured environment.