CoolFace
Apppublic

NA0XY/invoice-processing-env

sourceHugging Faceupdated 6mo agoView on Hugging Face
0likes
App README

InvoiceProcessingEnv

Round 1 submission: a real-world OpenEnv benchmark for accounts-payable operations.

This environment models finance workflows that humans perform daily:

  1. 1.invoice field extraction from noisy OCR
  2. 2.invoice to purchase-order validation
  3. 3.fraud triage in invoice batches
  4. 4.GL coding for expense line items
  5. 5.vendor statement reconciliation

It is designed for agent learning and evaluation through the standard OpenEnv API:

  1. 1.reset()
  2. 2.step(action)
  3. 3.state()

Round 1 Compliance Snapshot

RequirementStatusNotes
Real-world task (non-toy)PassAP automation domain used in production finance workflows
OpenEnv API + typed modelsPassPydantic Observation/Action/Reward + reset/step/state implemented
3+ tasks with gradersPass5 tasks, deterministic graders, reward range [0.0, 1.0]
Meaningful reward shapingPassDelta-based step rewards with partial progress signals
Baseline inference scriptPassRoot-level inference.py using OpenAI client and required env vars
HF Space + Docker deployablePassDockerized app on port 7860 and HF-ready metadata
Documentation completenessPassSpaces, tasks, setup, validation, and submission checks documented

Real-World Utility

The agent acts as an AP analyst assistant. Given invoices and accounting context, it must:

  1. 1.Extract structured fields (vendor, invoice number, date, line items, amounts)
  2. 2.Validate invoices against purchase orders — detect quantity/price mismatches
  3. 3.Detect fraud in a batch — duplicates, unauthorized vendors, inflated amounts
  4. 4.Assign GL codes for ambiguous expense line items
  5. 5.Reconcile statements against internal ledgers for missing/discrepant invoices

OpenEnv Interface

MethodEndpointBehavior
reset(task_id, optional custom body)POST /resetInitializes task state and returns first observation
step(action)POST /stepApplies one action and returns observation, reward, done, info
state()GET /stateReturns current episode state snapshot

All reward values are constrained to [0.0, 1.0].

Typed Observation Space

FieldTypeDescription
task_idstringActive task
task_descriptionstringWhat the agent must do
step_numberintCurrent step
total_stepsintEpisode length
invoiceobject{id, raw_text, metadata}
purchase_orderobject or nullPO for task 2
vendor_whitelistlist or nullApproved vendors for task 3
batchlist or nullAll 5 invoices for task 3
chart_of_accountsobject or nullGL mapping reference for task 4
vendor_statementstring or nullRaw statement text for task 5
internal_ledgerlist or nullInternal ledger entries for task 5

Typed Action Space

FieldRequired byValues
invoice_idAllstring
extracted_fieldsTask 1dict of 7 fields
decisionTasks 2 & 3approve / reject / flag_for_review
mismatchesTask 2list of strings
fraud_flagsTask 3list of {invoice_id, reason}
gl_allocationsTask 4object {line_item_description: GL-XXXX}
missing_invoicesTask 5list of invoice numbers
discrepancy_invoicesTask 5list of invoice numbers

Task Suite and Difficulty

TaskDifficultyMax StepsGoal
task_1Easy3Extract 7 fields from a noisy invoice
task_2Medium3Validate invoice vs PO, find 3 mismatches
task_3Hard3Detect 3 fraudulent invoices in a batch of 5
task_4Medium3Assign GL codes for 4 line items
task_5Hard3Reconcile statement vs ledger

Grader and Reward Design

Task 1 (extraction): score = correct_fields / 7 Each of the 7 fields contributes 1/7. Numeric fields tolerate ±$0.02 or ±2%.

Task 2 (validation): 0.4 × correct_decision + 0.2 × mismatches_found Max 1.0. Each of the 3 mismatches is detected via keyword matching.

Task 3 (fraud detection): F1(precision, recall) + 0.05 × correct_reasons F1 across fraud IDs plus a reason bonus. Max 1.0.

Task 4 (GL coding): 0.25 × correct_gl_assignment One quarter point per correctly coded line item. Max 1.0.

Task 5 (reconciliation): 0.5 × F1(missing) + 0.5 × F1(discrepancy) Balances missing-invoice detection and discrepancy detection. Max 1.0.

Environment step rewards are delta-based: each step returns only improvement over the best score seen so far in the episode.

Mandatory Inference Requirements

The root-level inference script is named inference.py and uses the OpenAI client.

Required environment variables:

  1. 1.APIBASEURL
  2. 2.MODEL_NAME
  3. 3.HF_TOKEN

Recommended additional variable for local runs:

  1. 1.ENVBASEURL

The script emits strict structured stdout logs:

  1. 1.[START]
  2. 2.[STEP]
  3. 3.[END]

The formatting is enforced by a compliance checker in tools/checkinferencelogs.py.

Setup

bash
# Run locally
pip install -r requirements.txt
uvicorn app.main:app --port 7860
bash
# Docker
docker build -t invoice-env .
docker run -p 7860:7860 invoice-env
bash
# Run baseline inference
export API_BASE_URL="https://api.openai.com/v1"
export MODEL_NAME="gpt-4o-mini"
export HF_TOKEN="sk-..."
export ENV_BASE_URL="http://localhost:7860"
python inference.py

API Endpoints

MethodPathDescription
GET/healthHealth check
GET/tasksList all tasks
POST/reset?task_id=task_1Start new episode
POST/stepSubmit action, get observation + reward
GET/stateRead current episode state

Baseline Scores

Baseline run is deterministic at temperature 0 (model-dependent absolute scores).

TaskScore
Task 1 — Field Extractiontask-dependent
Task 2 — PO Validationtask-dependent
Task 3 — Fraud Detectiontask-dependent
Task 4 — GL Codingtask-dependent
Task 5 — Reconciliationtask-dependent
Overallcomputed at runtime

Validation and Pre-Submission Checks

bash
pip install openenv-core
openenv validate
bash
# local container smoke
docker build -t invoice-env .
docker run -p 7860:7860 invoice-env
bash
# strict inference log compliance
python tools/check_inference_logs.py --stdout stdout_check.log --stderr stderr_check.log

For the full submission gate, run scripts/validate-submission.sh against your deployed Space URL.

Submission gate checklist:

  1. 1.HF Space deploy responds with 200 and reset works
  2. 2.OpenEnv validation passes
  3. 3.Docker build and run succeed
  4. 4.Inference script completes and emits strict log format
  5. 5.Task graders produce valid [0,1] rewards

Resource and Runtime Constraints

Target infra compatibility:

  1. 1.inference runtime under 20 minutes
  2. 2.compatible with vCPU=2 and memory=8GB

HF Space Deployment

This repository is configured for Docker Spaces.

  1. 1.Push to a Hugging Face Space with hardware cpu-basic
  2. 2.Ensure Space tag includes openenv
  3. 3.App listens on port 7860 via Dockerfile runtime

Judging Alignment (Round 1)

This submission is built to map directly to the published rubric:

  1. 1.Real-world utility: AP automation benchmark with realistic finance tasks
  2. 2.Task and grader quality: deterministic graders with easy to hard progression
  3. 3.Environment design: typed interfaces, delta rewards, clear episode boundaries
  4. 4.Code quality and compliance: OpenEnv-compatible structure + deployable container
  5. 5.Creativity and novelty: multi-stage AP flow in a single cohesive benchmark