CoolFace
Apppublic

ssanidhya0407/hackathon-openenv-100934

sourceHugging Faceupdated 6mo agoView on Hugging Face
0likes
App README

Support Triage OpenEnv

Support Triage OpenEnv is a real-world operations environment for evaluating agents on customer support workflows. The environment models the kind of work human support teams actually perform: reviewing inbound tickets, inspecting account context and internal evidence, routing cases to the right team, preparing customer replies, applying workflow tags, and closing tickets only after the case is operationally complete.

This is designed as a practical benchmark for agent evaluation rather than a toy game. The domain is enterprise SaaS support, with scenarios covering billing disputes, outages, shipping claims, account access incidents, spam handling, multilingual support, privacy requests, and large-value reconciliation cases.

The benchmark is intentionally structured around evidence-based triage. Agents are expected to inspect case context before closing tickets, use internal workflow tags to keep the queue organized, and handle different kinds of customer requests with the right operational posture.

To make the environment feel more like a real support desk than a static checklist, each ticket also carries a hidden SLA target and business-impact level. The environment surfaces that operational pressure through a live queue-health snapshot, so agents can see when overdue work, urgent containment, or backlog pressure are starting to affect the episode.

Why this environment is useful

  • —Current agent benchmarks rely heavily on basic web browsing or static QA datasets, leaving a real gap in operations-based workflow evaluation. This environment provides immediate value to the RL community by offering dense, trajectory-level rewards and deterministic grading in a highly demanded enterprise domain.
  • —The task family is operationally realistic: support organizations really do triage queues, escalate security incidents, reconcile invoices, manage privacy requests, and handle executive escalations.
  • —The environment is deterministic and reproducible, which makes it usable for benchmarking and regression testing.
  • —Episodes reward incremental progress rather than only terminal success, so it is suitable for both evaluation and learning.
  • —Tasks span easy, medium, and hard cases, including mixed queues where the agent must balance urgency across multiple tickets.
  • —The benchmark covers several high-value support subdomains in one environment, which makes it useful for comparing general-purpose agents rather than single-skill specialists.
  • —Hard tasks are not just larger queues: they include sequencing constraints, evidence review requirements, and cases where the wrong ticket handled first lowers the final score.

Core environment API

The environment implements the standard OpenEnv loop:

  • —reset(task_id: str | None) -> observation
  • —step(action) -> observation, reward, done, info
  • —state() -> state

The HTTP server exposes:

  • —POST /reset
  • —POST /step
  • —GET /state
  • —GET /tasks
  • —GET /grade
  • —GET /health

Action space

Agents act through the typed SupportAction model. Supported actions:

  • —inspect_ticket
  • —set_classification
  • —assign_owner
  • —add_tag
  • —update_history
  • —propose_resolution
  • —draft_reply
  • —close_ticket
  • —finish

These actions model a realistic support workflow rather than direct answer submission. In particular, inspect_ticket reveals internal evidence and customer context, while close_ticket enforces workflow completeness, required tagging, and evidence review.

Observation space

Each observation contains:

  • —task metadata (task_id, objective, step_count, max_steps)
  • —visible ticket queue state
  • —an operations_snapshot with queue pressure, overdue tickets, and the next SLA deadline
  • —internal notes uncovered through inspection
  • —evolving workflow state such as tags, assignment, and evidence review
  • —previous action result
  • —dense reward values and reward breakdown
  • —terminal status and structured info

Task bank

The environment includes a broad ticket bank across customer support subdomains:

  1. 1.easy_refund_single_ticket
  2. 2.medium_outage_and_shipping
  3. 3.hard_security_multi_ticket
  4. 4.expert_system_wide_crisis
  5. 5.vip_priority_escalation
  6. 6.subscription_loop_error
  7. 7.spanish_billing_query
  8. 8.technical_integration_bug
  9. 9.spam_ignore_case
  10. 10.bulk_corporate_refund

The three core benchmark tasks used by the baseline are the easy, medium, and hard scenarios. Additional tasks broaden the environment into multilingual support, privacy/compliance, enterprise escalation, and trust-and-safety handling.

Reward design

The reward function provides signal throughout the episode:

  • —evidence-gathering reward for reviewing the case before taking final action
  • —progress reward for first-time completion of meaningful workflow milestones
  • —queue-health shaping tied to SLA burn, urgent-ticket neglect, and backlog relief
  • —quality reward from the current grader projection
  • —penalties for invalid or incomplete actions
  • —terminal bonus tied to final case quality

This makes the environment useful for trajectory-level agent evaluation, not only pass/fail scoring.

Grading

Each task has deterministic hidden facts and a normalized 0.0-1.0 grader. The grader checks:

  • —correct case classification
  • —correct urgency assignment
  • —proper team routing
  • —appropriate resolution selection
  • —reply quality via expected operational content
  • —whether the agent reviewed available evidence before closing the case
  • —whether the agent touched the highest-risk case first when queue ordering matters
  • —required workflow tags
  • —closure completeness

The graders are programmatic, repeatable, and suitable for automated evaluation pipelines.

Setup

bash
python -m venv .venv
source .venv/bin/activate
pip install .

Run locally:

bash
uvicorn server.app:app --host 0.0.0.0 --port 7860

Baseline inference

The required root-level script is:

  • —inference.py

Required environment variables:

  • —API_BASE_URL
  • —MODEL_NAME
  • —HF_TOKEN

Run:

bash
export API_BASE_URL="https://api.openai.com/v1"
export MODEL_NAME="gpt-4o"
export HF_TOKEN="your_token_here"
python inference.py

Docker

bash
docker build -t openenv-support-triage .
docker run --rm -p 7860:7860 openenv-support-triage

The container starts a single-worker FastAPI process on port 7860 and exposes a built-in health endpoint at GET /health. This keeps the environment state deterministic across episodes and gives container platforms a simple readiness check.

Hugging Face Spaces

This repository is configured as a Docker Space and tagged for OpenEnv usage in the README metadata. The environment is intended to be run as a lightweight API-first benchmark on standard CPU hardware.

Recommended Space settings:

  • —SDK: Docker
  • —Hardware: CPU basic
  • —Port: 7860
  • —Public health probe: GET /health

Required Space variables:

  • —API_BASE_URL
  • —MODEL_NAME
  • —HF_TOKEN