HIMANSHUKUMARJHA/api-design-env
API Design RL Environment
A production-relevant OpenEnv environment where an AI agent learns to design REST APIs from functional requirements. Given a natural-language spec (e.g. "Design a REST API for an e-commerce product catalog with filtering, variants, and categories"), the agent submits endpoint definitions and receives multi-dimensional, partial-credit feedback until it converges on a correct design.
Motivation
API design is a routine engineering task with clear, objectively verifiable quality criteria: correct HTTP methods, RESTful path conventions, complete coverage of requirements, consistent naming, and proper status codes. Unlike toy tasks these properties map directly to real production code reviews, making this environment useful for training agents that assist with software engineering workflows.
Action Space
The agent submits an ApiDesignAction containing a list of EndpointSpec objects.
class EndpointSpec(Action):
method: str # HTTP method: GET, POST, PUT, PATCH, DELETE
path: str # URL path, e.g. /users/{id}/posts
description: str # What this endpoint does (default "")
request_body: dict # Request body schema {field: type} (default {})
response_body: dict # Response body schema {field: type} (default {})
status_code: int # Expected success status code (default 200)
query_params: list[str]# Supported query parameters (default [])
class ApiDesignAction(Action):
endpoints: list[EndpointSpec]Observation Space
After each step() the agent receives an ApiDesignObservation:
class ApiDesignObservation(Observation):
# Inherited from Observation base
done: bool # Episode finished?
reward: float | None # Shaped reward signal (0.0-1.2)
# Environment-specific
requirements: str # Natural-language problem description
constraints: list[str] # Specific constraints to satisfy
feedback: dict[str, float] | None# Per-dimension scores (5 axes)
suggestions: list[str] # Actionable improvement hints
attempt_number: int # Current attempt (1-indexed)
max_attempts: int # Maximum allowed attempts
total_score: float | None # Weighted total score (0.0-1.0)State
class ApiDesignState(State):
episode_id: str # Unique episode identifier
step_count: int # Steps taken so far
problem_id: str # Current problem identifier
difficulty: str # "easy" | "medium" | "hard"
best_score: float # Best total_score achieved this episode
max_attempts: int # Configured max attemptsReward Function
The reward is not binary. Every step() returns a composite signal built from five programmatic grading dimensions:
Partial progress signals:
- Each dimension scores 0.0-1.0 independently -- partial credit within each.
total_score = weighted sum(0.0-1.0).- Improvement bonus:
reward = total_score + 0.2 * max(0, score - prev_score). First good attempt gets a boost; subsequent regressions do not. - Penalty signals: empty submissions receive 0.0 across all dimensions. Endpoints with invalid HTTP methods, verbs in paths, or wrong status codes are penalised in the restfulconventions and httpsemantics dimensions.
The suggestions list provides natural-language hints (e.g. "Missing endpoint: DELETE /todos/{id}", "Avoid verbs in path: /getUser") that a language-model agent can use to iteratively improve.
Tasks (3 Difficulty Tiers, 11 Problems)
Easy (4 problems)
Medium (4 problems)
Hard (3 problems)
Every problem has a deterministic ground-truth solution. The grader compares the agent's submission against it structurally (path patterns, methods, schema fields) -- no LLM is needed for scoring.
Setup
Install
pip install openenv-core
git clone https://github.com/himanshu748/Scaler-hack.git
cd Scaler-hackRun locally (no Docker)
PYTHONPATH=. uvicorn api_design_env.server.app:app --host 0.0.0.0 --port 8000Run with Docker
docker build -t api-design-env -f api_design_env/Dockerfile api_design_env/
docker run -p 8000:8000 api-design-envValidate
cd api_design_env && openenv validate
# [OK] api_design: Ready for multi-mode deploymentDeploy to HF Spaces
cd api_design_env && openenv push --repo-id <username>/api-design-envUsage
Python (direct, no server)
from api_design_env import ApiDesignEnv, ApiDesignAction, EndpointSpec
from api_design_env.server.environment import ApiDesignEnvironment
env = ApiDesignEnvironment()
# Reset with difficulty or problem selection
obs = env.reset(seed=42, difficulty="easy")
# obs = env.reset(problem_id="todo_crud")
print(obs.requirements) # "Design a REST API for a simple todo-list..."
print(obs.constraints) # ["Support listing all todos...", ...]
# Submit a design
action = ApiDesignAction(endpoints=[
EndpointSpec(method="GET", path="/todos", description="List todos",
status_code=200, query_params=["completed", "limit"]),
EndpointSpec(method="POST", path="/todos", description="Create todo",
status_code=201, request_body={"title": "string"}),
])
obs = env.step(action)
print(obs.total_score) # 0.45 (partial credit)
print(obs.feedback) # {"completeness": 0.4, "restful_conventions": 1.0, ...}
print(obs.suggestions) # ["Missing endpoint: DELETE /todos/{id}", ...]WebSocket client (against running server)
from api_design_env import ApiDesignEnv, ApiDesignAction, EndpointSpec
with ApiDesignEnv(base_url="https://himanshukumarjha-api-design-env.hf.space").sync() as env:
result = env.reset()
print(result.observation.requirements)
result = env.step(ApiDesignAction(endpoints=[...]))
print(result.observation.total_score)Baseline Scores
Run the baseline evaluation (deterministic, seed=42):
python -m api_design_env # Pretty table
python -m api_design_env --json # Machine-readable JSONResults
- random -- generates structurally valid but content-irrelevant endpoints.
- heuristic -- parses requirement text to extract resource names and generates standard CRUD. No LLM.
- oracle -- submits the ground-truth solution (theoretical upper bound).
OpenAI baseline
export OPENAI_API_KEY=sk-...
python -m api_design_env.baseline_openaiUses gpt-4o-mini to read requirements and produce endpoint designs. See api_design_env/baseline_openai.py for details.
Endpoints (when deployed)
Project Structure
api_design_env/
├── __init__.py # Public exports
├── __main__.py # `python -m api_design_env` entry point
├── models.py # Action, Observation, State (Pydantic)
├── client.py # EnvClient subclass (WebSocket)
├── baseline.py # Heuristic + random + oracle baselines
├── baseline_openai.py # OpenAI API baseline (gpt-4o-mini)
├── openenv.yaml # OpenEnv manifest
├── pyproject.toml # Package metadata
├── Dockerfile # HF Spaces / production container
├── README.md # This file
└── server/
├── __init__.py
├── app.py # FastAPI application
├── environment.py # Core RL logic (reset/step/state)
├── grader.py # Multi-dimensional scoring engine
├── problems.py # 12 curated problems with ground truths
├── Dockerfile # Alternative Dockerfile
└── requirements.txt
tests/
└── test_api_design_env.py # 14 unit testsBuilt With
- OpenEnv by Meta & Hugging Face
- Built for the Meta PyTorch OpenEnv Hackathon x SST