skdoosh/debugging-mind-arena

DebuggingMindArena

DebuggingMindArena is a stateful OpenEnv debugging benchmark. Each episode presents one buggy Python function, exposes public test feedback after every patch attempt, and grades the patch with hidden tests inside a short-lived subprocess sandbox.

Quick Start

from firstenv import DebugAction, DebuggingMindArenaEnv

with DebuggingMindArenaEnv(base_url="http://localhost:8000") as env:
    result = env.reset(challenge_id="T0")
    print(result.observation.challenge_id)
    print(result.observation.code)

    result = env.step(
        DebugAction(
            code="def sum_to_n(n):\n    return sum(range(n + 1))\n",
            reasoning="Include n in the range",
        )
    )
    print(result.reward, result.done)
    print(result.observation.partial_test_results)

Environment Contract

Action

• —code: full candidate function definition
• —reasoning: optional explanation string

Observation

• —challenge_id
• —difficulty
• —code
• —log
• —partial_test_results
• —steps_taken
• —max_steps
• —standard OpenEnv done, reward, metadata

State

state() returns serializable episode state including current code, latest pass rates, and last error. Test callables are not exposed.

Challenge Set

• —T0: off-by-one summation
• —T1: palindrome slice mix-up
• —T2: empty-average contract returns None
• —T3: mutable default argument
• —T4: range upper-bound error in is_sorted
• —T5: Fibonacci returns the lagging variable

Reward

Each step uses:

difficulty_weight * overall_pass_rate - 0.05 * steps_taken

If the episode times out unsolved at max_steps, one extra difficulty weight is subtracted.

Local Development

Run the server:

uvicorn server.app:app --reload

Run tests:

pytest

Run the smoke baseline:

python inference.py