thrishaldevx/email-triage-env

📬 Email Triage OpenEnv

A real-world email inbox triage environment for training and evaluating AI agents. Built on the OpenEnv spec by Meta & Hugging Face.

Why Email Triage?

Email triage is one of the most common, high-value knowledge-worker tasks:

• —Knowledge workers spend 28% of their workweek managing email (McKinsey)
• —Enterprise AI assistants must handle priority classification, action routing, and professional reply drafting
• —Unlike games or coding tasks, email triage tests real-world language understanding under ambiguity

This environment fills a gap in the OpenEnv ecosystem: no existing environment covers natural language inbox management with graded reward signals across all three dimensions.

Environment Overview

Tasks

🟢 easy_triage — Priority Labeling

• —5 emails with clear, unambiguous priority signals
• —Agent must assign: urgent | high | medium | low
• —Max 10 steps
• —Expected score range: 0.7–1.0 for capable models

🟡 medium_triage — Priority + Action Selection

• —8 emails with mixed signals and some ambiguity
• —Agent must assign priority AND choose: reply | archive | delete | escalate
• —Max 16 steps
• —Expected score range: 0.45–0.75 for frontier models

🔴 hard_triage — Full Triage with Reply Drafting

• —12 emails across all categories
• —Agent must label priority, choose action, AND draft relevant professional replies for emails that require a response
• —Reply quality scored on: relevance, professionalism, urgency acknowledgment
• —Max 30 steps
• —Expected score range: 0.30–0.60 for frontier models

Action Space

{
  "email_id": "e001",
  "action_type": "label",
  "priority": "urgent",
  "reply_body": null
}

Action semantics:

• —label — assign a priority level to the email
• —reply — compose and send a reply (requires reply_body)
• —archive — file the email, no response needed
• —delete — remove from inbox (spam, newsletters, irrelevant)
• —escalate — flag as requiring immediate human attention
• —skip — pass (penalized: −0.05 reward)

Observation Space

{
  "current_email": {
    "id": "e001",
    "from": "cto@enterprise.com",
    "subject": "URGENT: Production database is down",
    "body": "...",
    "timestamp": "2024-03-15T09:02:00Z",
    "has_attachment": false,
    "labels": []
  },
  "inbox_summary": {
    "total": 5,
    "processed": 1,
    "pending": 4,
    "current_index": 1
  },
  "last_action_result": "Action 'label' recorded for email 'e001'. Progress: 1/5.",
  "last_action_error": null,
  "task_name": "easy_triage",
  "step_count": 1,
  "max_steps": 10,
  "score_so_far": 0.4,
  "done": false,
  "reward": 0.4
}

Reward Function

Rewards are dense — the agent receives signal at every step, not just at episode end.

step_reward = (
    priority_accuracy  × task_priority_weight  × 0.4
  + action_accuracy    × task_action_weight    × 0.4
  + reply_quality      × task_reply_weight     × 0.4
  − 0.05 × (action == "skip")
  − 0.10 × invalid_action
)

Priority accuracy: Full credit for exact match, partial credit for adjacent levels (e.g., predicting high when true is urgent gets 0.65 credit).

Action accuracy: Full credit for exact match, partial credit for semantically similar choices (e.g., reply when true is escalate gets 0.5 credit).

Reply quality (hard task): Heuristic scorer measuring:

• —Message length and substance (0.2)
• —Topic relevance — keyword overlap with subject/body (0.3)
• —Professional tone — greeting + sign-off (0.2)
• —Urgency acknowledgment for high-priority emails (0.3)

Task weights:

Final episode score = weighted average across all emails (0.0–1.0).

Setup & Usage

Prerequisites

• —Python 3.10+
• —Docker
• —pip install openenv-core

Local Development

# Clone and install
git clone https://huggingface.co/spaces/your-org/email-triage-env
cd email-triage-env
pip install -e .

# Run server locally
PYTHONPATH=. uvicorn email_triage_env.server.app:app --host 0.0.0.0 --port 7860

# In another terminal — quick smoke test
curl -s -X POST http://localhost:7860/reset \
  -H "Content-Type: application/json" \
  -d '{}' | python3 -m json.tool

Docker

# Build (from repo root)
docker build -f email_triage_env/server/Dockerfile -t email-triage-env .

# Run easy task
docker run -p 7860:7860 -e EMAIL_TRIAGE_TASK=easy_triage email-triage-env

# Run hard task
docker run -p 7860:7860 -e EMAIL_TRIAGE_TASK=hard_triage email-triage-env

# Test the server
curl -s http://localhost:7860/health
curl -s -X POST http://localhost:7860/reset -H "Content-Type: application/json" -d '{}'

Running the Baseline Inference Script

export HF_TOKEN=hf_your_token
export API_BASE_URL=https://router.huggingface.co/v1
export MODEL_NAME=Qwen/Qwen2.5-72B-Instruct
export IMAGE_NAME=email-triage-env   # local Docker image
export EMAIL_TRIAGE_TASK=easy_triage,medium_triage,hard_triage

python inference.py

Expected output:

[START] task=easy_triage env=email-triage-env model=Qwen/Qwen2.5-72B-Instruct
[STEP] step=1 action=label(e001,priority=urgent) reward=0.40 done=false error=null
[STEP] step=2 action=label(e010,priority=low) reward=0.40 done=false error=null
...
[END] success=true steps=5 score=0.820 rewards=0.40,0.40,0.40,0.40,0.00

OpenEnv Validation

openenv validate

Deploy to Hugging Face Spaces

# Install HF CLI
pip install huggingface_hub

# Login
huggingface-cli login

# Push
openenv push --repo-id your-org/email-triage-env

Baseline Scores

Tested with Qwen/Qwen2.5-72B-Instruct via HuggingFace router:

Frontier model ceiling (GPT-4o, Claude 3.5):

The hard task remains genuinely challenging: drafting relevant, professional replies that reference specific email context requires deep language understanding beyond surface-level classification.

Email Corpus

The environment includes 15 realistic email scenarios across 9 categories:

Each email has ground-truth labels for: true_priority, true_category, true_action, and requires_response.

Project Structure

email-triage-env/
├── __init__.py          # EmailTriageEnv, EmailAction, EmailObservation
├── models.py            # Pydantic model exports
├── client.py            # WebSocket EnvClient
├── inference.py         # ← Baseline agent (root level, per OpenEnv spec)
├── openenv.yaml         # Environment manifest
├── pyproject.toml       # Package config
├── README.md
└── server/
    ├── __init__.py
    ├── email_data.py    # 15 email scenarios + 3 task configs
    ├── environment.py   # EmailTriageEnvironment core logic
    ├── app.py           # FastAPI via openenv create_app()
    ├── requirements.txt
    └── Dockerfile

Environment Variables

License

MIT — contributions welcome.