Apppublic

Pisces3dragon/email-triage-env

sourceHugging Faceupdated 5mo agoView on Hugging Face

0likes

App README

Email Triage Environment

An OpenEnv environment that simulates a real-world email triage task. An AI agent must categorize, prioritize, and draft replies to incoming emails — a task that knowledge workers perform daily.

Motivation

Email triage is one of the most common yet cognitively demanding productivity tasks. Workers spend hours daily deciding which emails need immediate attention, which can wait, and which are spam. This environment tests an agent's ability to:

—Classify emails into categories (urgent, normal, follow_up, spam)
—Prioritize appropriately (1=highest to 5=lowest)
—Compose relevant replies when needed
—Avoid unnecessary replies to spam/informational emails

This is directly useful for training and evaluating agents that assist with email management.

Action Space

Field	Type	Description
`category`	string	One of: `urgent`, `normal`, `spam`, `follow_up`
`priority`	int	Priority level 1 (highest) to 5 (lowest)
`reply_draft`	string	Draft reply text (empty string if no reply needed)

Observation Space

Field	Type	Description
`email_id`	string	Unique email identifier
`sender`	string	Sender email address
`subject`	string	Email subject line
`body`	string	Full email body text
`timestamp`	string	ISO timestamp of when email was received
`feedback`	string	Feedback on the agent's last action
`emails_remaining`	int	How many emails are left in the session
`score_so_far`	float	Running average score in [0, 1]

Tasks

Easy (4 emails)

Obvious classification: clear spam (phishing, scam offers) vs clearly urgent (server down, direct orders from boss) vs routine HR announcements. Straightforward priority assignment.

Medium (5 emails)

Nuanced scenarios: client follow-ups with scheduling needs, security alerts requiring judgment, vendor support tickets, manager action items. Requires understanding context and urgency cues.

Hard (6 emails)

Complex multi-stakeholder situations: legal/NDA reviews with board meeting deadlines, budget reallocation requests needing VP approval, build failures blocking release pipelines, compliance deadlines with escalation threats. Requires synthesizing multiple signals and drafting substantive replies.

Reward Function

Each action is scored on three weighted components (total 0.0–1.0):

—Category accuracy (50%): Exact match with ground truth category
—Priority accuracy (30%): Full credit for exact match, partial credit for being off by 1 or 2
—Reply quality (20%): Based on whether a reply was needed and the substantiveness of the draft

The episode score is the average reward across all steps, normalized to [0, 1].

Baseline Scores

Using Qwen/Qwen2.5-72B-Instruct via HuggingFace Inference API:

Task	Expected Score Range
Easy	0.70 – 0.90
Medium	0.55 – 0.75
Hard	0.45 – 0.65

Setup & Usage

Prerequisites

—Python 3.10+
—Docker (for containerized deployment)
—pip install openenv-core

Run Locally

bash

# Install dependencies
pip install -e .

# Start the server
uvicorn server.app:app --host 0.0.0.0 --port 7860 --reload

# Or using uv
uv run server

Docker Build & Run

bash

cd server
docker build -t email-triage-env .
docker run -p 7860:7860 email-triage-env

Run Inference

bash

export API_BASE_URL="https://router.huggingface.co/v1"
export MODEL_NAME="Qwen/Qwen2.5-72B-Instruct"
export HF_TOKEN="your-hf-token"
export ENV_BASE_URL="http://localhost:7860"

python inference.py

Deploy to HF Spaces

bash

openenv push --repo-id your-username/email-triage-env

Environment Configuration

Required environment variables for inference:

—API_BASE_URL — LLM API endpoint
—MODEL_NAME — Model identifier
—HF_TOKEN — HuggingFace API key

Validation

bash

openenv validate