Freakdivi/HelpDesk

UPI Banking Support Environment

OpenEnv-style environment for evaluating agents on UPI customer support workflows. The benchmark focuses on realistic banking support decisions rather than generic FAQ matching.

Motivation

This environment is designed to test whether an agent can behave like a safe and useful support assistant for a UPI payments product such as Paytm, PhonePe, or Google Pay style support flows.

The goal is not only to answer customers correctly, but also to:

• —identify the right issue type
• —retrieve the right knowledge entry
• —escalate fraud or overdue review cases when needed
• —avoid unsafe behavior such as asking for PINs or OTPs
• —handle multi-turn conversations before closing a case

Environment Description

The environment uses three tasks with increasing difficulty:

• —easy: classify a customer issue into the correct support track
• —medium: choose the right FAQ or escalate when human/manual review is required
• —hard: run a short multi-turn support conversation with clarification, guidance, and closure

The current support tracks are:

• —payment_failure
• —refund_delay
• —fraud_complaint
• —kyc_account_restriction
• —upi_pin_or_bank_linking

The dataset includes:

• —10 banking FAQ entries in knowledge_base.json
• —10 easy tickets in easy.json
• —10 medium tickets in medium.json
• —10 hard tickets in hard.json

Action Space

The public baseline and server currently accept the legacy action names below, which are internally mapped to the compact action model in models.py.

Internally, these are normalized to:

• —ask_for_details
• —take_action
• —respond_to_user
• —escalate_case
• —close_case

Observation Space

The model receives an Observation object from models.py.

Important evaluation detail:

• —hidden gold labels such as the correct FAQ id and escalation label are not exposed to the model in the observation

Reward

Rewards are normalized to the range 0.0 to 1.0 in environment.py.

The final reward is shaped rather than purely binary. It combines:

• —correctness
• —safety
• —resolution
• —efficiency
• —penalties

Weighted reward:

0.35 * correctness
+ 0.30 * safety
+ 0.20 * resolution
+ 0.15 * efficiency
+ penalties

Examples:

• —correct classification gives a strong easy reward
• —correct FAQ retrieval gives partial progress on medium
• —correct escalation gives reward on medium
• —clarification plus guidance plus successful closure raises hard reward
• —unsafe prompts such as asking for PIN or OTP reduce reward sharply

Task Difficulty

Setup

From the package root:

cd /path/to/helpdesk_env
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt

Usage

Run Tests

cd /path/to/helpdesk_env
.venv/bin/python -m py_compile environment.py inference.py models.py

Run the Server

cd /path/to
PYTHONPATH=. /path/to/helpdesk_env/.venv/bin/uvicorn helpdesk_env.server.app:app --host 127.0.0.1 --port 8000

Build the Docker Image

cd /path/to/helpdesk_env
docker build -t helpdesk-openenv .
docker run --rm -p 8000:8000 helpdesk-openenv

Use the Python Client

from helpdesk_env.client import HelpdeskEnvClient

client = HelpdeskEnvClient("http://127.0.0.1:8000")
result = client.reset("easy")
print(result.observation.customer_message)

Run Inference

cd /path/to/helpdesk_env
export GROQ_API_KEY=your_key
.venv/bin/python inference.py

Optional model override:

export LLM_MODEL=llama-3.1-8b-instant
export TASK_NAME=medium

Baseline Scores

Latest observed Groq baseline run after removing answer leakage from the observation:

Interpretation:

• —easy is still quite direct and can be near-perfect for strong LLMs
• —medium and hard are more informative because they require retrieval, escalation judgment, and multi-turn behavior

Project Structure

helpdesk_env/
├── README.md
├── Dockerfile
├── .gitignore
├── .dockerignore
├── __init__.py
├── client.py
├── data/
│   ├── knowledge_base.json
│   └── tickets/
│       ├── easy.json
│       ├── medium.json
│       └── hard.json
├── environment.py
├── inference.py
├── models.py
├── openenv.yaml
├── requirements.txt
├── graders/
│   ├── category_grader.py
│   ├── faq_grader.py
│   └── resolution_grader.py
└── server/
    ├── app.py
    └── helpdesk_environment.py