paruljaintechie/email-triage-openenv

Email Triage OpenEnv

A real-world email triage environment where an AI agent must classify, prioritize, route, and respond to corporate emails. Built to the OpenEnv spec for training and evaluating LLM agents on genuine productivity tasks.

Motivation

Every knowledge worker spends hours daily triaging email. This environment models that workflow: the agent receives emails one at a time from a mixed corporate inbox (support tickets, executive escalations, billing disputes, partnership proposals, spam) and must make fast, accurate triage decisions. It tests classification accuracy, priority judgment, routing knowledge, action item extraction, and communication skills — all in one environment.

Architecture

Agent → receives email (Observation)
     → outputs triage decision (Action)
     → receives score + next email (Reward + Observation)
     → repeats until inbox is empty (done=True)

OpenEnv API

Observation Space

Each observation provides the agent with:

Action Space

The agent outputs a structured JSON action:

Tasks

Task 1: Email Classification (Easy)

Classify each email by category and detect sender sentiment. 10 emails per episode. Graded on category accuracy (65% weight) and sentiment detection (35% weight). Partial credit for related categories (e.g., urgent vs support).

Task 2: Priority Assignment & Routing (Medium)

All of Task 1, plus assign priority level (P0-P3) and route to the correct department. 10 emails. Graded across 4 dimensions with heavier weight on priority (30%) and routing (30%). Extra penalty for under-prioritizing critical P0 items. Bonus for perfect scores.

Task 3: Full Email Triage with Draft Response (Hard)

Complete triage: classify, prioritize, route, extract action items, and draft an appropriate reply. 12 emails. Graded across 6 dimensions including action item coverage and reply quality. Tests the full range of agent capabilities.

Expected difficulty:

• —Easy task: frontier models should score 0.80+
• —Medium task: frontier models should score 0.65-0.80
• —Hard task: frontier models should score 0.50-0.70

Reward Function

Rewards are computed per-step (per-email) with values in [0.0, 1.0]:

• —Partial credit: Close-but-wrong answers earn partial scores (e.g., classifying urgent as support earns 0.4 instead of 0.0; priority off by one level earns 0.5)
• —Graduated penalties: Under-prioritizing critical items (calling a P0 "P3") is penalized more heavily than over-prioritizing. Replying to spam is penalized. Missing follow-up flags costs points.
• —Bonuses: Perfect scores on all dimensions earn a 5% bonus.
• —Deterministic: Same inputs always produce the same scores.

Environment Variables (MANDATORY)

Before running inference, set these environment variables:

export API_BASE_URL="https://router.huggingface.co/v1"   # LLM API endpoint
export MODEL_NAME="Qwen/Qwen2.5-72B-Instruct"            # Model identifier
export HF_TOKEN="hf_..."                                   # Your HF / API key

Setup & Usage

Install Dependencies

pip install -r requirements.txt

Run Inference Script (MANDATORY — produces structured stdout logs)

# With API key:
export HF_TOKEN="hf_..."
python inference.py

# Without API key (mock agent for testing):
python inference.py

The inference script emits structured stdout in the required format:

[START] task=task_classify env=email-triage model=Qwen/Qwen2.5-72B-Instruct
[STEP] step=1 action=cat=spam|sent=neutral|pri=P3|dept=spam_filter reward=1.00 done=false error=null
[STEP] step=2 action=cat=urgent|sent=negative|pri=P0|dept=engineering reward=0.83 done=false error=null
...
[END] success=true steps=10 score=0.835 rewards=1.00,0.83,...

Run Validation

python validate.py

Run Baseline (detailed output)

python baseline.py --mock           # Mock mode (no API key)
python baseline.py --model gpt-4o   # Live mode

Start HTTP Server

python server.py
# Server runs at http://localhost:7860

Docker

docker build -t email-triage-openenv .
docker run -p 7860:7860 email-triage-openenv

HTTP API Examples

# Health check
curl http://localhost:7860/health

# List tasks
curl http://localhost:7860/tasks

# Start episode (empty body defaults to task_classify)
curl -X POST http://localhost:7860/reset \
  -H "Content-Type: application/json" -d '{}'

# Start specific task
curl -X POST http://localhost:7860/reset \
  -H "Content-Type: application/json" \
  -d '{"task_id": "task_full_triage"}'

# Submit action
curl -X POST http://localhost:7860/step \
  -H "Content-Type: application/json" \
  -d '{"session_id": "abc123", "action": {"email_id": "...", "category": "support", "sentiment": "negative", "priority": "P1", "department": "support"}}'

Baseline Scores

Hugging Face Deployment

1. 1.Create a new HF Space (Docker type)
2. 2.Upload all files to the Space repo
3. 3.Add the tag openenv to the Space metadata
4. 4.The Space will auto-build from the Dockerfile and serve on port 7860
5. 5.Verify: curl -X POST -d '{}' -H 'Content-Type: application/json' https://your-space.hf.space/reset

Pre-Submission Checklist

• —[ ] python validate.py — all checks pass
• —[ ] python inference.py — completes without error, emits [START]/[STEP]/[END] format
• —[ ] docker build -t email-triage-openenv . — builds successfully
• —[ ] docker run -p 7860:7860 email-triage-openenv — starts and responds to /reset
• —[ ] HF Space deployed and responding to POST /reset with {} body
• —[ ] openenv.yaml present with 3+ tasks
• —[ ] All rewards in 0.0-1.0 range
• —[ ] Graders are deterministic and reproducible

Project Structure

email-triage-openenv/
├── openenv.yaml              # OpenEnv metadata and config
├── models.py                 # Typed models (Observation, Action, Reward, StepResult, EnvState)
├── email_triage_env.py       # Core environment (step/reset/state/close)
├── server.py                 # HTTP server (FastAPI + stdlib fallback)
├── inference.py              # MANDATORY inference script with structured stdout
├── baseline.py               # Detailed baseline script (optional, for development)
├── validate.py               # Spec validation script
├── Dockerfile                # Container (vcpu=2, 8GB optimized)
├── requirements.txt          # Python dependencies
├── data/
│   ├── __init__.py
│   └── email_generator.py    # Synthetic email dataset with ground truth
├── graders/
│   ├── __init__.py
│   ├── grader_utils.py       # Shared scoring (partial credit, penalties, bonuses)
│   ├── classify_grader.py    # Task 1 grader (easy)
│   ├── prioritize_route_grader.py  # Task 2 grader (medium)
│   └── full_triage_grader.py # Task 3 grader (hard)
└── results/                  # Output directory

License

MIT