thirunagarisairamcharan/email-inbox-manager

Email Inbox Manager (OpenEnv)

1. Overview

Email Inbox Manager is an OpenEnv-style benchmark in which an AI agent operates a simulated email inbox: it reads messages, replies, composes mail, applies labels, deletes messages, and searches—under explicit task instructions and step budgets. The environment is deterministic given a PRNG seed, exposes typed observations and actions over HTTP, and scores behavior with dense, interpretable rewards suitable for evaluating tool use, planning, and instruction following in a realistic productivity domain without live mail systems or third-party APIs during rollouts.

2. Environment description

3. Action space

Discriminated union on `action_type` (email_env.models.Action). JSON uses snake_case; `message_id` identifies messages.

4. Observation space

Returned on reset (inside ResetResponse.observation) and inside `StepResult.observation` after each step.

Each EmailMessage includes: message_id, subject, body, from_address, to_addresses, cc_addresses, bcc_addresses, date_sent, read, thread_id, in_reply_to, folder, importance, has_attachments, snippet, metadata.

5. Tasks

Episode success is commonly taken as final holistic score = 1.0 before or at max steps.

6. Reward function

• —After every step the environment recomputes a holistic score h ∈ [0,1] via `get_grader(task_id)(GraderContext)` (deterministic from inbox + sent + initial snapshot).
• —`StepResult.reward` is the step-wise delta: Δh = h_new − h_prev (clamped for API typing), yielding a dense signal whenever progress increases.
• —`Observation` / `StepResult.info` exposes `holistic_score` and `cumulative_reward` (sum of per-step deltas) for logging.
• —Partial credit is explicit: easy has a 0.5 read-only tier; medium decomposes into three weighted fractions; hard uses 0.2 / 0.5 / 1.0 tiers before a qualifying outbound message.
• —Episode ends when h = 1.0 (success) or steps ≥ max_steps (timeout).

7. Setup instructions

Local (pip + uvicorn)

python -m venv .venv
# Windows: .venv\Scripts\activate
source .venv/bin/activate   # Linux/macOS

pip install -r requirements.txt
uvicorn main:app --host 0.0.0.0 --port 7860

Requires Python 3.11+ (Dockerfile targets 3.11).

Docker

docker build -t email-inbox-manager .
docker run --rm -p 7860:7860 email-inbox-manager

Image listens on 7860; HEALTHCHECK hits `GET /health`.

8. Running inference

1. 1.Start the API (see §7).
2. 2.Set LLM router / OpenAI-compatible variables, then run:

export MODEL_NAME="your-model-id"
export HF_TOKEN="hf_..."
export API_BASE_URL="https://router.huggingface.co/v1"   # optional
python inference.py

The script loops `easy_reply` → `medium_triage` → `hard_thread`, calls `POST /reset` and `POST /step` via httpx, and queries the LLM with the OpenAI client. It prints per-task scores and a final summary table.

9. OpenEnv compliance

• —Typed contract: Pydantic v2 models for Observation, Action (discriminated union), InboxState, ResetResponse, StepResult.
• —Stateful episodes: EmailEnv + `SESSION_STORE` keyed by `session_id`; thread-safe locks per session.
• —Core API surface: `POST /reset`, `POST /step`, `GET /state/{session_id}`, `GET /tasks`, `GET /health`.
• —Task registry: `TASK_REGISTRY` / `TaskConfig` (id, difficulty, max_steps, description).
• —Manifest: `openenv.yaml` documents metadata, API, observation/action space, tasks, reward intent, Docker.
• —Reproducibility: `seed` on reset + deterministic `generate_inbox` and graders.
• —Containerization: Production-oriented Dockerfile (slim base, layer caching, non-root user, healthcheck, port 7860).

10. Baseline scores (placeholder)

Representative mean holistic score at max_steps (fill in after official eval runs):

Replace TBD with aggregate metrics (mean, std, success@1.0) once benchmarks are frozen.

Project layout

Environment name: email-inbox-manager — version aligned with `openenv.yaml`.