vsdjr-reddy/ticket-triage-openenv

🎫 Ticket Triage — OpenEnv Environment

A real-world customer support ticket triage environment for training and evaluating AI agents. Agents must read incoming support tickets and produce structured triage decisions: severity, category, routing team, and a human-readable internal summary.

This is a task that support engineers at every SaaS company perform hundreds of times daily. Getting it wrong delays resolution, misroutes work, and burns customer trust.

Environment Description

The environment presents support tickets (single, batch, or escalation chain) and expects the agent to output structured TriageDecision objects for each ticket.

Why this domain?

• —Every software company has a support queue
• —Triage quality directly affects customer satisfaction and churn
• —The task requires reading comprehension, urgency assessment, business judgment, and structured output — a genuine challenge for language agents
• —Partial credit is natural: getting severity right but routing wrong is better than total failure

Action & Observation Spaces

Observation

{
  "task_id": "easy_single_ticket",
  "step": 0,
  "tickets": [
    {
      "ticket_id": "TKT-001",
      "subject": "Cannot login — password reset not working",
      "body": "...",
      "customer_tier": "premium",
      "created_at": "2024-01-15T10:00:00Z",
      "related_ticket_ids": []
    }
  ],
  "instructions": "Triage the following support ticket...",
  "previous_actions": [],
  "feedback": null,
  "done": false
}

Action

{
  "decisions": [
    {
      "ticket_id": "TKT-001",
      "severity": "high",
      "category": "account",
      "assigned_team": "tier2_support",
      "summary": "Premium customer locked out with 2-hour deadline...",
      "is_duplicate_of": null,
      "merge_with": []
    }
  ]
}

Severity options: low | medium | high | critical

Category options: billing | technical | account | feature_request | security | general

Team options: tier1_support | tier2_support | billing_team | engineering | security_team | account_management | executive_escalation

Tasks

Task 1 — easy_single_ticket (Easy)

Objective: Triage a single, unambiguous support ticket.

A premium customer reports being unable to reset their password with an urgent deadline. The correct answer requires recognising time pressure (HIGH severity), identifying this as an account issue, and routing to tier2_support.

Grader: Checks severity (30%), category (30%), team (25%), summary quality (15%).

Expected difficulty: A well-prompted GPT-4o-mini should score ≥ 0.85.

Task 2 — medium_batch_triage (Medium)

Objective: Process a batch of 5 mixed tickets simultaneously.

Tickets span all categories:

• —A billing overcharge (HIGH, billing team)
• —A dark-mode feature request (LOW, tier1)
• —A production API outage at an enterprise customer (CRITICAL, engineering)
• —A suspicious foreign login (HIGH/CRITICAL, security team)
• —A how-to question (LOW, tier1)

Grader: Partial credit per ticket (mean of 5 scores). Missing tickets score 0.

Expected difficulty: Requires correct multi-label classification. A distracted agent that misses security implications will score ~0.50.

Task 3 — hard_escalation_chain (Hard)

Objective: Manage a 4-ticket escalation thread from a single enterprise customer.

Three tickets (TKT-201, 202, 203) form an escalating thread: initial report → "nobody responded" escalation → legal threat. A fourth ticket (TKT-204) is a coincidentally related but independent report from a different standard-tier customer.

The agent must:

1. 1.Identify TKT-201 as canonical; mark 202 and 203 as duplicates
2. 2.Assign CRITICAL severity and route to engineering or executive_escalation
3. 3.Write an executive-level summary covering timeline, revenue impact, and SLA risk
4. 4.Handle TKT-204 independently as LOW severity

Grader: Checks canonical routing (30%), duplicate detection (20%), executive summary quality incl. business-impact keywords (20%), independent TKT-204 handling (15%), SLA awareness (15%).

Expected difficulty: Frontier models (GPT-4o) score ~0.70–0.85. Smaller models often fail duplicate detection or write superficial summaries.

Reward Function

Rewards are dense (not sparse):

• —Each sub-dimension (severity, category, team, summary) contributes independently
• —Partial credit: e.g., getting severity right but team wrong still scores ~0.30–0.55
• —Summary quality is scored on word count and absence of boilerplate
• —Hard task: business-impact keywords in the executive summary provide additional signal
• —Episode ends when max_steps is reached or score ≥ 0.95 (early termination on near-perfect)

This design means a random baseline scores ~0.10 and an optimal agent scores ~0.90–1.0, giving ample gradient signal over the full trajectory.

API Endpoints

Setup & Usage

Run with Docker

docker build -t ticket-triage-openenv .
docker run -p 7860:7860 ticket-triage-openenv

Run locally

pip install -r requirements.txt
uvicorn server:app --host 0.0.0.0 --port 7860

Run tests

pytest tests/ -v

Run baseline inference

export API_BASE_URL="https://api.openai.com/v1"
export MODEL_NAME="gpt-4o-mini"
export HF_TOKEN="sk-..."
export TRIAGE_ENV_URL="http://localhost:7860"

python inference.py

Baseline Scores

Scores from gpt-4o-mini at temperature=0:

Project Structure

.
├── openenv.yaml          # OpenEnv metadata
├── server.py             # FastAPI server (reset/step/state endpoints)
├── inference.py          # Baseline inference script
├── requirements.txt
├── Dockerfile
├── README.md
├── env/
│   ├── __init__.py
│   ├── models.py         # Typed Pydantic models (Observation, Action, Reward, ...)
│   └── environment.py    # TicketTriageEnv core class
└── tasks/
    ├── __init__.py
    └── definitions.py    # 3 tasks + deterministic graders