rushilgara/product-prioritization-env

Product Feature Prioritization OpenEnv

What the environment does

An agentic OpenEnv where AI agents must learn to prioritize which product features to build next. Agents evaluate unstructured customer feedback signals, business metrics (impact/effort), technical debt, and CEO/stakeholder pressure to determine the highest-ROI work items, acting as an AI Product Manager.

Example task

Task 3: The Messy "Hard" Task You have capacity to start exactly ONE feature this sprint. You are given an unstructured feed of conflicting signals:

• —[Eng] VP of Eng: 'If we don't start Database Sharding Phase 1 (High Effort) now, the site will definitively crash during Black Friday in 3 months. I will quit if we delay this again.'
• —[Sales] Head of Sales: 'We have a $5M enterprise contract waiting to be signed. They will drop us if we don't deliver SAML SSO (Medium Effort) before the CEO's Q3 earnings call NEXT WEEK.'
• —[Support] CS Lead: 'Search latency is a bleeding neck. We lost 30 SMB customers this month (total $1M ARR lost) because search takes 15 seconds. Effort to fix is Low.'
• —[Exec] Board Member: 'Every competitor has an AI Copilot (High Effort). We need one immediately to look innovative.'

Agent's Goal: Weigh these inputs, calculate the immediate vs long-term tradeoffs across Churn Risk ($1M), Business Value ($5M), Technical Debt (3-month deadline), and Stakeholder Pressure (CEO/Board), and select the most critical feature (SAML SSO) to build next. Provide a JSON justification defending the choice.

Action and Observation Spaces

Observation Space:

• —task_level (string): The current difficulty level (Easy, Medium, Hard).
• —instruction (string): Specific directions for what the agent must do.
• —context (string): Unstructured background context (customer complaints, executive pressure, metrics).
• —features (list[FeatureRequest]): Available features to prioritize, each containing id, title, description, and optionally user_requests, impact_score, and effort_score.

Action Space:

• —selected_feature_id (string, optional): ID of the single feature prioritized (used in Easy and Hard).
• —ranked_feature_ids (list[string], optional): Ranked feature IDs from highest to lowest priority (used in Medium).
• —justification (string, optional): The agent's textual rationale to defend the product decision (Required for Hard).

Reward explanation

Rewards range from 0.0 to 1.0 depending on nuance (partial logic scaling):

• —Task 1 (Partial Logic): 1.0 for choosing the critical app crash bug, 0.5 for choosing a high-demand but non-critical feature (Apple Auth), 0.0 for choosing low-impact cosmetic changes.
• —Task 2 (Distance Scaling): Ranking 5 items. The optimal ROI ranking is calculated based on business rules. The reward is linearly scaled down based on exactly how far the agent's ranking deviated from the optimal list.
• —Task 3 (Nuanced Logic):
• —1.0 for perfect identification (SAML SSO) + sound business mathematical justification ($5M won vs $1M lost).
• —0.8 for identifying the correct $5M feature but missing the mathematical tradeoff logic in the text.
• —0.7 for picking Search Latency ($1M churn saved) - Good PM focus, but mathematically suboptimal compared to $5M.
• —0.5 for picking Database Sharding - Noble long-term engineering focus, but sacrificed immediate revenue and missed impending deadlines.
• —0.0 for picking the AI Copilot - Caved to board pressure with zero user demand or ROI.

How to run locally

First, install the required dependencies:

pip install -r requirements.txt

Start the standard OpenEnv FastAPI server locally:

uvicorn server.app:app --host 0.0.0.0 --port 7860

In a new terminal, run the baseline evaluation script against it:

export API_BASE_URL="https://api.openai.com/v1"
export MODEL_NAME="gpt-4o-mini"
export HF_TOKEN="your-openai-api-key"
python inference.py

How to deploy

This environment is inherently compliant with Hugging Face Spaces out-of-the-box.

1. 1.Create a new Docker Space on Hugging Face.
2. 2.Upload this entire repository directory.
3. 3.Hugging Face will automatically use the included Dockerfile to build the required python:3.11-slim image and automatically expose uvicorn on port 7860.
4. 4.Tag your space with openenv so the evaluators can discover it.

Example API request/response

POST /step Request:

{
  "action": {
    "selected_feature_id": "feat_y",
    "justification": "Fixing the exact-match search bug resolves immediate enterprise churn, saving immediate active revenue."
  }
}

Response:

{
  "observation": {
    "task_level": "Completed",
    "instruction": "All tasks completed.",
    "context": "",
    "features": [],
    "metadata": {
      "reason": "Perfect prioritization. Selected the feature with highest customer pain (immediate enterprise churn risk) and provided sound business justification.",
      "step": 3
    }
  },
  "reward": 1.0,
  "done": true
}

Baseline results table

Because of the distance-based ranking in Task 2 and the contextual nuance required in Task 3, frontier models do not automatically score 1.0 perfectly without robust logic.