Anicher/email_triage

📧 Email Triage OpenEnv

A real-world email triage and prioritization environment for training and evaluating AI agents. Agents must process an inbox of emails — classifying urgency, routing to departments, drafting replies, flagging escalations, and archiving spam.

Why Email Triage?

Knowledge workers spend 2–3 hours per day on email. This is one of the most common, highest-value tasks for LLM agents to assist with — yet no standardized benchmark exists. This environment fills that gap.

Environment Description

The agent receives an inbox of realistic business emails spanning:

• —Security incidents and production outages
• —Customer support requests and complaints
• —Sales inquiries and contract negotiations
• —HR matters (leave, reviews, complaints)
• —Legal and billing urgencies
• —Spam and low-priority marketing

The agent must take structured actions on each email to process the inbox effectively.

Action Space

Each action targets one email and specifies one operation:

Observation Space

{
  "inbox": [{"id", "subject", "sender", "body", "timestamp", "thread_id"}],
  "processed": ["e001", "e003", ...],
  "step_count": 5,
  "task_id": "task_easy",
  "score_so_far": 0.45,
  "message": "Classification: urgency=1.0, category=0.5"
}

Tasks

Task 1 — Email Classification (Easy)

• —Emails: 10
• —Max steps: 30
• —Goal: Classify each email by urgency (urgent/normal/low) and category
• —Score: Mean of per-email classification accuracy
• —Expected difficulty: Frontier models ~0.80+, smaller models ~0.50-0.65

Task 2 — Email Routing & Reply Drafting (Medium)

• —Emails: 15
• —Max steps: 60
• —Goal: Route emails to the correct department AND draft replies for actionable emails
• —Score: 40% routing + 40% reply quality + 20% coverage
• —Expected difficulty: Frontier models ~0.65-0.75, smaller models ~0.40-0.55

Task 3 — Inbox Zero Sprint (Hard)

• —Emails: 20
• —Max steps: 100
• —Goal: Full pipeline — classify, route, reply, escalate, archive spam
• —Score: 25% classify + 25% routing + 25% reply + 15% escalation + 10% spam handling
• —Expected difficulty: Frontier models ~0.55-0.70, smaller models ~0.30-0.45

Reward Function

The environment provides dense reward signals — not just end-of-episode scores:

• —Correct classification: +1/n_emails × (0.5×urgency + 0.5×category)
• —Correct routing: +1/n_emails × department_match
• —Quality reply: +1/n_emails × keyword_overlap_score
• —Archive spam/low-priority: +0.8/n_emails
• —Correct escalation: +1/n_emails
• —Archive urgent email: -0.5/n_emails ← penalty
• —Reply to spam: -0.3/n_emails ← penalty
• —False escalation: -0.3/n_emails ← penalty
• —Invalid action: -0.05

API

Reset

POST /reset
{"task_id": "task_easy", "session_id": "my_session"}

Step

POST /step
{
  "session_id": "my_session",
  "action": {
    "email_id": "e001",
    "action_type": "classify",
    "urgency": "urgent",
    "category": "technical"
  }
}

State

GET /state?session_id=my_session

Final Score

GET /score?session_id=my_session

Setup & Usage

Local (Docker)

docker build -t email-triage-env .
docker run -p 7860:7860 email-triage-env

Local (Python)

pip install -r requirements.txt
python main.py

Inference

export HF_TOKEN=your_key
export API_BASE_URL=https://router.huggingface.co/v1
export MODEL_NAME=Qwen/Qwen2.5-72B-Instruct
export ENV_BASE_URL=http://localhost:7860

python inference.py

Baseline Scores

Measured with Qwen/Qwen2.5-72B-Instruct via HuggingFace Inference API:

Project Structure

email-triage-env/
├── app/
│   ├── __init__.py
│   ├── email_data.py      # 20 emails with ground-truth labels
│   ├── environment.py     # EmailTriageEnv (step/reset/state)
│   ├── graders.py         # Per-task scoring functions
│   ├── models.py          # Pydantic typed models
│   └── server.py          # FastAPI server
├── tests/
│   └── test_environment.py
├── Dockerfile
├── inference.py           # Baseline agent script
├── main.py
├── openenv.yaml
├── requirements.txt
└── README.md

License

MIT