razak123/code-migration-env

Code Migration Environment

code_migration_env is an OpenEnv benchmark for a real engineering workflow: migrating production code while preserving behavior, modernizing idioms, and meeting operational constraints.

Instead of toy puzzles, the environment evaluates the kind of work platform and application engineers actually do:

• —modernizing old Python utilities before a runtime upgrade
• —porting an API integration from Python to Node.js
• —rewriting a memory-sensitive analytics pipeline from pandas to Polars

The benchmark is designed to reward more than surface-level syntax changes. Agents must preserve function behavior, satisfy deterministic tests, and adopt the target ecosystem's idioms.

Why This Is Useful

Code migration is a frequent and expensive engineering task. Teams regularly:

• —update shared utilities during Python version upgrades
• —move service clients across languages during platform consolidation
• —replace slow or memory-heavy data pipelines with faster libraries

This environment packages those workflows into reproducible tasks with programmatic grading so model quality can be compared reliably.

Action Space

Observation Space

Tasks

Each task includes:

• —a source implementation
• —a concrete migration objective
• —hidden deterministic checks
• —idiom-sensitive grading
• —scenario-specific context such as stakeholder constraints and pitfalls

Reward Design

The reward function gives partial credit instead of only binary success.

• —syntax credit for code that parses in the target language
• —functional credit when deterministic tests pass
• —idiom credit for using migration-specific target patterns
• —a small efficiency penalty for needing extra repair attempts

To satisfy OpenEnv validation and downstream evaluation rules, task rewards are clamped to the open interval (0, 1).

Tasks now support iterative refinement across multiple attempts. When a submission is syntactically valid but still fails tests or misses important idioms, the environment returns actionable feedback in history and allows the agent to revise its solution before the episode ends.

Baseline Behavior

The included `inference.py` uses the OpenAI client and reads model settings from environment variables. A typical baseline run with mistralai/devstral-2-123b-instruct-2512 produces scores in roughly this range:

These values can vary slightly with model behavior and proxy settings, but they should stay within the valid OpenEnv score range.

Setup

Build and run the environment locally:

docker build -t code-migration-env .
docker run -p 8000:8000 code-migration-env

Run the baseline agent:

export API_BASE_URL=https://your-litellm-proxy.example/v1
export MODEL_NAME=mistralai/devstral-2-123b-instruct-2512
export API_KEY=your_proxy_api_key
export HF_TOKEN=your_hf_token_if_needed
python inference.py

Validate the environment structure:

openenv validate

Resource Expectations

The environment is designed to run comfortably on modest infrastructure:

• —CPU-only execution
• —no local LLM weights loaded into the container
• —small deterministic test fixtures
• —suitable for runners around 2 vCPU / 8 GB RAM

Most runtime cost comes from remote LLM latency during baseline inference, not from the FastAPI environment itself.

Environment Design Notes

• —Episodes are intentionally short and lightweight, but no longer purely single-shot. Agents can improve over successive attempts using structured feedback.
• —The public state is limited to progress metadata and task context. Hidden solutions and raw grader internals are not exposed there.

Repository Structure

.
├── client.py
├── inference.py
├── models.py
├── openenv.yaml
├── scenarios/
│   ├── easy/
│   ├── medium/
│   └── hard/
└── server/
    ├── app.py
    └── code_migration_env_environment.py

Notes For Reviewers

Two design choices are intentional:

• —The benchmark stays deterministic and lightweight enough for automated evaluation.
• —The task observations include operational context so agents are rewarded for migrations that feel production-aware, not just syntactically valid.