Hashirama01/finops-env

1---2title: FinOps Support Automation3emoji: 💰4colorFrom: blue5colorTo: green6sdk: docker7pinned: false8app_port: 80009base_path: /web10tags:11  - openenv12  - agent13  - customer-support14  - tool-use15  - finops16---17 18# FinOps Support Automation — OpenEnv Environment19 20An OpenEnv environment where an LLM agent resolves real customer support21tickets by navigating a mock billing and CRM backend. Built for the22**Meta PyTorch × HuggingFace OpenEnv Hackathon (Round 1)**.23 24> **Tagline**: Agents train on the same tasks a Tier-2 billing support engineer does every day — search a customer, inspect their invoices, issue a targeted refund, apply a loyalty discount, cancel a subscription — with deterministic graders and dense reward shaping.25 26---27 28## Why this environment29 30Most existing OpenEnv environments simulate games (Atari, Sudoku, Wordle), simple REPLs, or generic coding tasks. Real enterprise LLM value — the workloads companies actually pay for today — lives in agents that resolve support tickets, navigate internal APIs, and handle billing operations with care.31 32**FinOps Support Automation fills that gap.** It models tier-2 billing/CRM support as a structured tool-use task: each episode begins with a natural-language customer ticket, and the agent must resolve it by calling a fixed catalog of tools over a mock in-memory database. No browser simulation, no HTML parsing, no LLM-judge non-determinism — pure text in, tool-calls out, deterministic state diffs for grading.33 34### Differentiation from prior work35 36The closest academic benchmark is Sierra AI's **τ-bench** (retail / airline customer support). This environment differentiates along three dimensions:37 381. **Multi-system correlation.** Task 3 requires joining across two independent namespaces (CRM ↔ Billing) — no single-API lookup solves it.392. **Dense per-step reward shaping.** τ-bench is terminal 0/1. FinOps emits a shaped reward on every step (step cost, invalid-action penalty, destructive-regression penalty, terminal grader score).403. **Destructive-action asymmetry.** Wrong refunds and wrong cancellations are penalized twice as heavily as missed actions — an RL-friendly encoding of "don't touch what you don't understand."41 42---43 44## Environment specification45 46### Observation space (`FinopsObservation`)47 48| Field              | Type                     | Description                                                       |49|--------------------|--------------------------|-------------------------------------------------------------------|50| `task_id`          | `str`                    | `task1` / `task2` / `task3`                                       |51| `ticket`           | `str`                    | Natural-language customer support request                        |52| `last_response`    | `dict[str, Any]`         | JSON result of the previous tool call (or episode-start message) |53| `available_tools`  | `list[dict]`             | Tool catalog: name, description, argument schema                  |54| `steps_taken`      | `int`                    | Actions executed so far                                           |55| `max_steps`        | `int`                    | Episode budget                                                    |56| `done`             | `bool`                   | Episode-terminated flag (inherited from base `Observation`)       |57| `reward`           | `float`                  | Per-step shaped reward, can be negative (inherited)               |58| `metadata`         | `dict[str, Any]`         | Contains `cumulative_reward` and `task_id` (inherited)            |59 60### Action space (`FinopsAction`)61 62A single tool invocation:63 64```json65{"tool": "search_customers", "args": {"name": "TechCorp"}}66```67 68`tool` must be one of the 11 names in the catalog; `args` is a JSON object of keyword arguments.69 70### Tool catalog71 72| Tool                                   | Destructive | Purpose                                             |73|----------------------------------------|:---:|-----------------------------------------------------|74| `search_customers(name)`               |     | Fuzzy search customers by name substring            |75| `get_customer(customer_id)`            |     | Fetch one customer by id                            |76| `list_customers(status="active")`      |     | List customers filtered by status                   |77| `list_invoices(customer_id)`           |     | All invoices for a given customer                   |78| `get_invoice(invoice_id)`              |     | Fetch one invoice by id                             |79| `list_crm_users(status)`               |     | List CRM-side users by status (Task 3)              |80| `get_subscription_by_email(email)`     |     | Look up billing subscription by email (Task 3)      |81| `create_refund(invoice_id)`            |  ✔  | Refund an invoice — mutates DB                      |82| `apply_discount(customer_id, percent)` |  ✔  | Apply a % discount — mutates DB                     |83| `cancel_subscription(subscription_id)` |  ✔  | Cancel a billing subscription — mutates DB          |84| `submit()`                             |     | Signal task complete; ends the episode              |85 86### Reward shaping87 88Per-step reward is the sum of:89 90| Event                                              | Reward           |91|---------------------------------------------------|------------------|92| Per-step time cost                                 | `-0.01`          |93| Invalid tool name / bad args                       | `-0.10`          |94| Destructive call that *decreased* the grader score | `-0.50`          |95| Terminal grader score on `submit` or timeout       | `+(0.02, 0.98)`  |96 97The destructive-regression penalty is computed by re-grading the DB before and after each mutating call — if the score drops, the agent gets the `-0.5` penalty in addition to the step cost. This creates a dense, informative training signal and actively discourages destructive guessing.98 99**Why the strict `(0, 1)` range?** All grader outputs are passed through a monotone linear squish — `0.02 + raw · 0.96` — so no task ever reports exactly `0.0` or exactly `1.0`. This satisfies the hackathon's deep validator (which requires `score ∈ (0, 1)` strictly) while preserving ordering, so the destructive-regression detection and all F1 comparisons continue to work exactly as before.100 101### State (`State`)102 103Standard OpenEnv `State` with `episode_id` and `step_count`. Exposed at `GET /state`.104 105### Endpoints106 107Auto-generated by `openenv.core.env_server.http_server.create_app`:108 109- `POST /reset` — start a new episode (round-robins through task1 → task2 → task3 if no `task_id` specified)110- `POST /step` — execute one `FinopsAction`111- `GET /state` — current `State`112- `GET /schema` — machine-readable action/observation schemas113- `WS /ws` — persistent WebSocket session for efficient multi-step episodes114 115---116 117## The 3 tasks118 119### Task 1 — Easy: Relational Lookup & Refund120 121> *"Customer 'TechCorp Inc' emailed saying they were double-charged for their March invoice. Please find the duplicate charge and refund exactly one of the two identical March invoices. Do not touch any other customer's invoices."*122 123**Seed DB**: 15 customers, 18 invoices. TechCorp has two identical $500 March charges (`inv_901`, `inv_902`). Two other customers (Initech, Stark Industries) also have March duplicates as wrong-customer traps — refunding their invoices scores zero.124 125**Optimal policy (~4 steps)**:1261. `search_customers(name="TechCorp")` → `cus_001`1272. `list_invoices(customer_id="cus_001")` → sees two identical March charges1283. `create_refund(invoice_id="inv_902")` → refund one of them1294. `submit` → end130 131**Grader** (raw → reported after `(0,1)` squish):132- raw `1.0` → **`0.98`** — exactly one of `{inv_901, inv_902}` refunded, others untouched133- raw `0.5` → **`0.50`** — both TechCorp March duplicates refunded (over-refunded)134- raw `0.0` → **`0.02`** — wrong customer refunded or no refund at all135 136**Budget**: `max_steps = 10`137 138---139 140### Task 2 — Medium: Aggregation & Bulk Action141 142> *"Apply a 10% loyalty discount to every active customer whose lifetime spend (sum of all their invoices) exceeds $500. Do NOT discount customers below the threshold."*143 144**Seed DB**: 8 active + 2 inactive customers, 16 invoices. Six active customers qualify (`cus_001` $900, `cus_003` $1200, `cus_005` $750, `cus_007` $2000, `cus_012` $1500, `cus_018` $850). Traps: `cus_010` at exactly $500 (boundary, excluded), `cus_008` near-boundary distractor ($450), and two inactive customers with high spend who must not receive discounts.145 146**Optimal policy (~16 steps)**:1471. `list_customers(status="active")` → all 8 active customers1482. For each customer: `list_invoices(customer_id=...)`, sum amounts1493. For each qualifier: `apply_discount(customer_id=..., percent=10)`1504. `submit`151 152**Grader**: F1 score over the set of discounted `customer_id`s vs. the golden set `{cus_001, cus_003, cus_005, cus_007, cus_012, cus_018}`. If any applied discount is not 10%, the score is halved. Final output is squished into `(0.02, 0.98)` — a perfect run reports `0.98`, a total miss reports `0.02`.153 154**Budget**: `max_steps = 20`155 156---157 158### Task 3 — Hard: Multi-System Correlation159 160> *"Cross-reference our CRM with our Billing system. For any user whose CRM status is 'cancelled' but who still has an 'active' billing subscription, cancel the billing subscription. Do NOT cancel subscriptions that are already cancelled."*161 162**Seed DB**: Two disjoint namespaces.163- CRM: 13 users. Eight have `status=cancelled` (`crm_b`, `crm_d`, `crm_e`, `crm_g`, `crm_i`, `crm_k`, `crm_m`, `crm_o`).164- Billing: 12 subscriptions keyed by email. Three are already cancelled (`sub_205`, `sub_209`, `sub_213`).165 166**Traps**: Three CRM-cancelled users whose billing subs are already cancelled (must NOT re-cancel). One dead-end: `crm_o` (olga@ex.com) has no billing subscription at all.167 168**Correct target set**: `{sub_202 (bob), sub_204 (dave), sub_207 (grace), sub_211 (karen)}`.169 170**Optimal policy (~14 steps)**:1711. `list_crm_users(status="cancelled")` → 8 cancelled users1722. For each: `get_subscription_by_email(email=...)` → inspect billing status1733. Cancel only those with current status `active` → 4 cancellations1744. `submit`175 176**Grader**: Weighted F1 over the set of newly-cancelled subscription IDs. False positives (wrong cancellations) cost **2×** more than false negatives (missed cancellations). Pre-existing cancellations (`sub_205`, `sub_209`, `sub_213`) are filtered out so they don't inflate precision. Final output is squished into `(0.02, 0.98)` — a perfect trap-avoided run reports `0.98`.177 178**Budget**: `max_steps = 20`179 180---181 182## Baseline scores183 184Run with the provided `inference.py` using an OpenAI-compatible client. The full 3-task suite completes in well under the 20-minute / 2-vCPU / 8-GB hackathon runtime budget.185 186Scores below are the squished grader outputs — `0.98` is the achievable maximum (corresponding to a raw score of `1.0`) and `0.02` is the floor (raw `0.0`). This keeps every reported score strictly within the open interval `(0, 1)`.187 188| Model                              | Provider            | Task 1 | Task 2 | Task 3 | Mean | Runtime |189|------------------------------------|---------------------|-------:|-------:|-------:|-----:|--------:|190| `llama-3.3-70b-versatile`          | Groq                | **0.980** | 0.790 | **0.980** | **0.917** | 266 s |191| *scripted optimal reference*       | deterministic agent | 0.980 | 0.980 | 0.980 | 0.980 | — |192 193### What the scores mean194 195- **Task 1 (easy) — 0.980**: The agent solved the duplicate-refund task in 4 steps (`search_customers` → `list_invoices` → `create_refund` → `submit`), navigating past two wrong-customer duplicate traps (Initech and Stark Industries) to correctly target TechCorp's invoices. `0.980` is the highest possible reported score (raw 1.0, squished).196- **Task 2 (medium) — 0.790**: The agent identified 4 of 6 qualifying customers, missing `cus_012` and `cus_018`. It correctly excluded the boundary trap (`cus_010` at exactly $500) and the near-boundary distractor (`cus_008` at $450). The partial F1 score of ~0.80 squishes to 0.79 — demonstrating the task's meaningful difficulty gradient.197- **Task 3 (hard) — 0.980**: The agent correctly cross-referenced 13 CRM users ↔ 12 billing subscriptions, identified 8 CRM-cancelled users, inspected each, cancelled only the 4 with active billing subs (`sub_202`, `sub_204`, `sub_207`, `sub_211`), avoided 3 already-cancelled traps and 1 dead-end (no matching sub). This is the strongest signal that the destructive-action asymmetry penalty is training meaningful caution.198 199The scripted-reference row is not an LLM score — it's generated by invoking the same `step()` code path a perfect agent would take, establishing the achievable upper bound and confirming all three graders return `0.98` (raw `1.0`) on correct trajectories.200 201---202 203## Quickstart204 205### Run locally (no Docker)206 207```bash208# from project root209uv sync          # creates .venv and installs all dependencies210 211# start the server212uvicorn server.app:app --host 0.0.0.0 --port 8000213```214 215### Run the baseline inference script216 217```bash218# From project root — defaults to Groq + llama-3.3-70b-versatile219export API_BASE_URL="https://api.groq.com/openai/v1"  # any OpenAI-compatible endpoint220export MODEL_NAME="llama-3.3-70b-versatile"            # or gpt-4o-mini, Qwen2.5-7B-Instruct, etc.221export HF_TOKEN="gsk_..."                              # your API key (Groq/OpenAI/HF/etc.)222 223python3 inference.py224```225 226The script runs all 3 tasks end-to-end and emits structured `[START]` / `[STEP]` / `[END]` logs in the format required by the hackathon spec. Total runtime stays well under the 20-minute / 2-vCPU / 8-GB budget because each episode is bounded by `max_steps` (8 / 25 / 25) and the in-memory DB means tool calls are essentially free.227 228### Run the container229 230```bash231# from project root232docker build -t finops-env:latest .233docker run --rm -p 8000:8000 finops-env:latest234# curl -X POST http://localhost:8000/reset235```236 237### Validate spec compliance238 239```bash240openenv validate .241# [OK] finops_env: Ready for multi-mode deployment242```243 244---245 246## File layout247 248```249.250├── openenv.yaml             # Manifest (spec_version, runtime, port)251├── pyproject.toml           # Package metadata + dependencies252├── uv.lock                  # Pinned dependency graph253├── Dockerfile               # Container build for HF Spaces deployment254├── README.md                # (this file)255├── inference.py             # Baseline LLM agent loop (hackathon-required)256├── validator.bash           # Runs openenv validate against a live server257├── .env.example             # Template for required environment variables258├── __init__.py259├── models.py                # FinopsAction, FinopsObservation (Pydantic)260├── client.py                # EnvClient subclass for remote agents261├── tools.py                 # 11 tool functions + TOOL_CATALOG262├── server/263│   ├── app.py               # FastAPI app via openenv.core.http_server264│   ├── finops_env_environment.py  # FinopsEnvironment — reset/step/state265│   └── __init__.py266├── tasks/267│   ├── __init__.py          # TaskConfig dataclasses wiring tickets → seeds → graders268│   ├── database.py          # Per-task seed DBs (tasks 1–3)269│   └── graders.py           # Deterministic task graders (F1, weighted F1)270├── assets/                  # Static assets (graphs, diagrams)271└── outputs/                 # Inference run outputs272```273 274---275 276## License277 278Submitted under the same BSD-style license as the OpenEnv project templates.279