althafali/email-triage-env

📧 Email Triage Environment

An OpenEnv environment simulating real-world email triage — where an AI agent processes an inbox, classifying emails by urgency, setting priorities, archiving spam, and drafting professional replies.

  

🎯 Motivation

Email overload is one of the most common productivity challenges in professional settings. The average knowledge worker receives 120+ emails per day and spends ~2.5 hours reading and responding. An AI agent that can intelligently triage emails — sorting by urgency, filtering spam, and even drafting responses — could save hours of human effort daily.

This environment provides a standardized testbed for evaluating how well AI agents handle this complex, multi-faceted task.

🏗️ Environment Description

The Email Triage Environment presents an AI agent with a simulated inbox of emails. The agent processes emails one at a time through the standard OpenEnv step() / reset() / state() API.

How It Works

1. 1.`reset()` — Initializes a new episode, loads the email dataset, returns the first email
2. 2.`step(action)` — Agent takes an action on the current email; receives the next email + reward
3. 3.`state()` — Returns episode metadata (progress, score, missed urgents, etc.)

Each email comes with ground-truth labels. The environment scores the agent's actions using a partial-credit reward function that rewards close guesses and penalizes critical mistakes (like missing an urgent email).

📋 Action Space

👁️ Observation Space

📝 Tasks

Task 1: Spam Detection (Easy)

• —Emails: 20
• —Time Limit: 2 minutes
• —Actions: classify only
• —Goal: Correctly identify spam vs. legitimate emails
• —Scoring: Binary accuracy (spam / not-spam)

Task 2: Multi-Label Categorization (Medium)

• —Emails: 30
• —Time Limit: 3 minutes
• —Actions: classify, flag, archive
• —Goal: Classify emails into 6 categories AND assign correct priority
• —Scoring: 60% category accuracy + 40% priority accuracy (with partial credit)

Task 3: Full Triage + Response Drafting (Hard)

• —Emails: 40
• —Time Limit: 5 minutes
• —Actions: classify, reply, flag, archive, escalate
• —Goal: Full professional email triage — classify, prioritize, draft replies, handle urgents
• —Scoring: 40% category + 20% priority + 30% response quality + 10% urgency handling

🏆 Reward Function

Rewards provide partial credit throughout the episode (not just binary end scores):

🚀 Setup & Usage

Prerequisites

• —Python 3.10+
• —pip

Installation

# Clone the repository
git clone https://github.com/althafalimohommad/email-triage-env.git
cd email-triage-env

# Install dependencies
pip install -e .

Running the Server

# Start the environment server
uvicorn server.app:app --host 0.0.0.0 --port 8000

# Or run directly
python -m server.app

Running the Inference Script

# Set required env variables
export HF_TOKEN="hf_xxx"                 # Linux/Mac
$env:HF_TOKEN = "hf_xxx"                 # Windows PowerShell

# Run against local server
python inference.py

# Run against deployed HF Space
$env:ENV_URL = "https://althafali-email-triage-env.hf.space"
python inference.py

# Use a custom model
$env:MODEL_NAME = "meta-llama/Llama-3.3-70B-Instruct"
python inference.py

Using the Client

from email_triage_env import EmailTriageEnv, EmailTriageAction

with EmailTriageEnv(base_url="http://localhost:8000") as client:
    result = client.reset()
    print(f"First email: {result.observation.subject}")

    action = EmailTriageAction(
        action_type="classify",
        category="spam",
        priority=1,
        reason="Phishing attempt"
    )
    result = client.step(action)
    print(f"Reward: {result.reward}")

Docker

# Build
docker build -f server/Dockerfile -t email-triage-env .

# Run
docker run -p 8000:8000 email-triage-env

📊 Baseline Scores

Scores are approximate and may vary slightly between runs.

🚀 Deploying to Hugging Face Spaces

This environment can be deployed as a containerized Hugging Face Space (required for submission).

Option A — Using the OpenEnv CLI (recommended)

# Install the CLI
pip install openenv-core

# Login to HF
huggingface-cli login

# Push to your HF account (tags the space with 'openenv' automatically)
openenv push --repo-id YOUR_HF_USERNAME/email-triage-env

Option B — Manual Docker Build

# Build locally
docker build -f server/Dockerfile -t email-triage-env .

# Test locally before pushing
docker run -p 8000:8000 email-triage-env

# Then create a Space at huggingface.co/new-space (type: Docker)
# and push with git

Tag your HF Space with openenv so it appears in the hackathon's leaderboard.

📁 Project Structure

email_triage_env/
├── openenv.yaml              # Environment manifest
├── pyproject.toml             # Python dependencies
├── models.py                  # Pydantic models (Action, Observation, State)
├── client.py                  # WebSocket client
├── __init__.py                # Package exports
├── data/
│   ├── emails_easy.json       # 20 emails for spam detection
│   ├── emails_medium.json     # 30 emails for categorization
│   └── emails_hard.json       # 40 emails for full triage
├── server/
│   ├── app.py                 # FastAPI application
│   ├── email_triage_env_environment.py  # Core environment logic
│   ├── email_generator.py     # Synthetic data generator
│   └── Dockerfile             # Container config
├── tasks/
│   ├── task_easy.py           # Easy task grader
│   ├── task_medium.py         # Medium task grader
│   └── task_hard.py           # Hard task grader
├── baseline/
│   └── inference.py           # Baseline LLM agent
└── README.md                  # This file

🔒 License

BSD License — see LICENSE for details.