CoolFace
Apppublic

Jabbastin/ratingspace

sourceHugging Faceupdated 6mo agoView on Hugging Face
0likes
App README

App Store Review Prioritization Environment

This repository contains an OpenEnv-compatible environment that simulates a real-world App Store Review Prioritization system. The goal of this environment is to train and evaluate an AI agent that ranks user reviews based on urgency and importance for developers.

Real-World Motivation

App developers are often inundated with a high volume of user reviews. It's crucial to quickly identify and address the most critical feedback, such as bug reports about crashes, payment failures, or security issues. Manually triaging these reviews is time-consuming and inefficient. This environment provides a platform to develop and test AI agents that can automate this prioritization process, enabling development teams to be more responsive and improve user satisfaction.

Observation Space

The observation space provides the agent with the context needed to make a ranking decision.

  • —app_name (string): The name of the application.
  • —reviews (list of objects): A list of user reviews.
  • —id (string): A unique identifier for the review.
  • —text (string): The content of the user review.

Example

json
{
  "app_name": "Payment App",
  "reviews": [
    {"id": "r1", "text": "App crashes during checkout"},
    {"id": "r2", "text": "Nice UI"},
    {"id": "r3", "text": "Money deducted but order failed"}
  ]
}

Action Space

The agent's action is to provide a ranked list of review IDs.

  • —ranking (list of strings): A list of all review IDs from the observation, ordered from most important to least important.

Constraints

  • —The list must contain every review ID from the observation exactly once.
  • —There should be no duplicate or missing IDs.

Tasks

The environment includes three tasks of increasing difficulty:

  • —Easy: 3-5 reviews with very clear differences in severity (e.g., a critical crash vs. positive feedback).
  • —Medium: 6-10 reviews with a mix of severities and some ambiguity. The agent needs to distinguish between major and minor bugs.
  • —Hard: 10-15 reviews featuring subtle issues, ambiguous language, and multiple reviews of similar high severity. This challenges the agent's ability to make fine-grained distinctions.

Setup and Usage

Local Setup

  1. 1.Clone the repository:
bash
    git clone <repository-url>
    cd <repository-name>
  1. 1.Create and activate a virtual environment:
bash
    python -m venv venv
    source venv/bin/activate
  1. 1.Install dependencies:
bash
    pip install -r requirements.txt
  1. 1.Run the FastAPI server:
bash
    uvicorn app.main:app --host 0.0.0.0 --port 7860

Docker

  1. 1.Build the Docker image:
bash
    docker build -t app-store-review-env .
  1. 1.Run the Docker container:
bash
    docker run -p 7860:7860 app-store-review-env

Running Inference

The inference.py script provides a baseline for interacting with the environment using an OpenAI-compatible model.

  1. 1.Set up your model endpoint: Make sure you have a local or remote LLM server running that is compatible with the OpenAI API. You can use tools like Ollama or a hosted service.
  1. 1.Configure environment variables: Create a .env file in the root of the project and add the following:
    API_BASE_URL=http://<your-model-host>:<port>/v1
    MODEL_NAME=<your-model-name>
    OPENAI_API_KEY=<your-api-key>

Alternatively, you can export these variables in your shell.

  1. 1.Run the inference script:
bash
    python inference.py

Example Output (from inference.py)

The inference script will output structured logs for each task.

[START]
[STEP]
task: easy
query: Photo Editor Pro
ranking: ['r1', 'r3', 'r2']
reward: 0.95...
ndcg: 0.99...
precision_at_k: 0.99...
mrr: 0.99...

[STEP]
task: medium
query: Secure Wallet
ranking: ['r1', 'r6', 'r2', 'r3', 'r4', 'r5']
reward: 0.92...
ndcg: 0.98...
precision_at_k: 0.99...
mrr: 0.99...

...

[END]