paradox44/mutationgym-env

MutationGym (OpenEnv)

MutationGym is an OpenEnv environment that evaluates an agent's ability to write pytest tests that catch buggy implementations ("mutants") without failing the reference solution.

Quick start (local)

pip install -e .
python -m mutationgym_env.server.app

Then in another shell:

from mutationgym_env.models import MutationGymAction
from mutationgym_env.client import MutationGymEnv

env = MutationGymEnv(env_url="http://localhost:8000")
obs = env.reset()
tests = """
from solution import clamp

def test_basic():
    assert clamp(-1, 0, 10) == 0
    assert clamp(5, 0, 10) == 5
    assert clamp(11, 0, 10) == 10
"""
result = env.step(MutationGymAction(tests_py=tests, finalize=True))
print(result.reward, result.killed, result.total_mutants, result.error)

Docker

docker build -t mutationgym-env .
docker run --rm -p 8000:8000 mutationgym-env

Baseline evaluation

python examples/baseline_eval.py

Green agent wrapper

python green_agent/run_eval.py

Training

See training/README.md and training/grpo_train.py for a minimal GRPO run.

Environment design

• —Action: a single pytest file plus a finalize flag (optionally include task_id and seed for stateless HTTP calls).
• —Observation: task prompt, kill counts, reference pass/fail, runtime, and surviving mutant ids.
• —Reward: mutant kill rate minus penalties for false positives, runtime, and extra steps.

Task authoring

Each task lives in mutationgym_env/tasks/task_*.json and includes:

• —id, signature, prompt
• —reference implementation (list of lines or string)
• —mutants list with id and code

Add a new task file, then rebuild the container.

Safety limits

• —Max test file size (20 KB)
• —Per-run timeout (2s) on pytest executions
• —Forbidden imports and dangerous builtins (os, sys, subprocess, eval, exec, open)

OpenEnv workflow

openenv build
openenv validate --verbose
openenv push