SayujGupta2005/gst-reconciliation-env

GST Reconciliation OpenEnv

A real-world OpenEnv benchmark for GST invoice reconciliation. The environment simulates a finance operations workflow where an AI agent receives OCR-like invoice text, extracts key fields, validates the correct GST tax regime, and flags deterministic arithmetic discrepancies before submission.

The benchmark is intentionally text-first rather than image-OCR-first. This is a deliberate design choice: it models a genuine back-office finance task while remaining fully deterministic, dependency-light, and reproducible on free open-source tooling — no proprietary OCR or paid APIs required.

Why this domain

GST invoice verification is a high-volume back-office task in Indian finance operations. Teams routinely check invoice metadata, taxable amounts, interstate vs intrastate tax logic (IGST vs CGST+SGST), and arithmetic consistency before accepting invoices into ERP or Input Tax Credit (ITC) workflows. Errors — whether accidental or fraudulent — have direct compliance and cash-flow consequences.

This environment turns that real process into a clean, deterministic agent benchmark with dense intermediate rewards and a meaningful episode-level objective.

Benchmark design

The environment implements the standard OpenEnv interface: typed Action, Observation, and State models, plus reset(), step(), and a state property.

Observation space

Action space

The agent sends exactly one typed action per step:

Tasks

1. Basic extraction — easy

Goal: Extract the five mandatory invoice fields required by the workflow.

Required fields: supplier_gstin, invoice_number, invoice_date, taxable_value, total_invoice_value

Grader: Exact-match fraction over the five required fields. Score = correct / 5, in [0.0, 1.0].

2. Tax validation — medium

Goal: Identify the correct GST tax regime and validate the tax breakup.

Required reasoning:

• —Parse taxable value and GST rate from invoice text
• —Compare the first two digits of the supplier GSTIN with the place-of-supply state code
• —If they match → intrastate → CGST_SGST; if they differ → interstate → IGST
• —Extract the correct tax amounts for the determined regime

Grader: Weighted partial credit:

Maximum score: 1.0. A wrong correct_tax_type caps the score at 0.50.

3. Fraud detection — hard

Goal: Detect an arithmetic discrepancy in the invoice and formally flag it before submitting.

Required reasoning:

• —Compute expected tax from taxable_value × gst_rate / 100
• —Infer the correct tax regime from GSTIN state codes
• —Compare expected_total_invoice_value with stated_total_invoice_value
• —Call flag_discrepancy with the correct discrepancy_type, line_item, and a rationale that contains the key numeric evidence

Grader: Two independent components:

If the agent never calls flag_discrepancy, the episode score is capped at the extraction subtotal (≤ 0.30).

Reward shaping

The environment emits dense per-step rewards rather than a single sparse terminal signal:

This gives useful intermediate learning signal while preserving a meaningful episode-level objective.

Determinism

Each call to reset() advances a round-robin index over the scenario list in a fixed curriculum:

easy_001 → medium_001 → hard_001 → easy_001 → ...

Every run is fully reproducible and easy to validate locally before submission.

Project structure

.
├── Dockerfile
├── README.md
├── client.py                         # Top-level re-export of GSTReconciliationEnv
├── inference.py                      # Baseline hybrid agent (heuristic + LLM fallback)
├── models.py                         # Top-level re-export of GSTAction / GSTObservation / GSTState
├── openenv.yaml                      # OpenEnv spec manifest
├── pyproject.toml                    # Package build config
├── requirements.txt                  # Pinned runtime dependencies
├── validate-submission.sh            # Local pre-submission validation helper
├── server
│   ├── __init__.py
│   ├── app.py                        # Thin shim — delegates to gst_reconciliation_env
│   └── environment.py                # Thin shim — delegates to gst_reconciliation_env
└── gst_reconciliation_env
    ├── __init__.py                   # Package exports
    ├── client.py                     # Typed OpenEnv HTTP client
    ├── models.py                     # GSTAction, GSTObservation, GSTState (Pydantic v2)
    └── server
        ├── __init__.py
        ├── app.py                    # FastAPI app via create_fastapi_app
        ├── environment.py            # GSTReconciliationEnvironment — step / reset / grading logic
        └── scenarios.py              # SCENARIOS list (one fixed scenario per difficulty level)

Setup

python -m venv .venv
source .venv/bin/activate
pip install -e .

Start the server locally:

uvicorn gst_reconciliation_env.server.app:app --host 0.0.0.0 --port 7860 --reload

Health / connectivity check:

# GET the root page (always 200 OK)
curl http://localhost:7860/

# Or POST to /reset directly
curl -X POST http://localhost:7860/reset \
  -H "Content-Type: application/json" -d '{}'

Docker

Build and run:

docker build -t gst_reconciliation:latest .
docker run --rm -p 7860:7860 gst_reconciliation:latest

OpenEnv validation

openenv validate

Or use the bundled helper script, which pings a live Space, builds Docker locally, and runs openenv validate:

bash validate-submission.sh https://<your-space>.hf.space .

Baseline inference

The baseline uses a hybrid deterministic agent: regex-based extraction and arithmetic run first; an OpenAI-compatible chat model is called only when the heuristic cannot determine the next action. This keeps runs cheap and reproducible while satisfying the requirement to use the OpenAI client.

Environment variables

Run against a local server

export ENV_BASE_URL=http://localhost:7860
export API_BASE_URL=https://router.huggingface.co/v1
export MODEL_NAME=Qwen/Qwen2.5-72B-Instruct
export HF_TOKEN=your_token
python inference.py

Run against a local Docker image

export LOCAL_IMAGE_NAME=gst_reconciliation:latest
export API_BASE_URL=https://router.huggingface.co/v1
export MODEL_NAME=Qwen/Qwen2.5-72B-Instruct
export HF_TOKEN=your_token
python inference.py

Note on Docker unavailability: If LOCAL_IMAGE_NAME is set but Docker is not reachable in the runtime (e.g. the OpenEnv validator sandbox), inference.py logs a warning and automatically falls back to the HTTP client pointed at ENV_BASE_URL. The process always exits with code 0 — no unhandled exceptions propagate to the validator.

Expected baseline scores

Because the heuristic agent uses deterministic regex patterns tuned to the included scenarios, it should score near ceiling on all three tasks when the environment is running correctly:

Exact scores depend on the OpenEnv runtime version and any future scenario additions.

Hugging Face Spaces deployment

openenv push --repo-id <your-username>/gst-reconciliation-openenv

After deployment, confirm the Space is live:

curl -X POST https://<your-space>.hf.space/reset \
  -H "Content-Type: application/json" -d '{}'

Free and open-source stack

• —Python 3.10+
• —FastAPI — async REST server
• —Uvicorn — ASGI server
• —Pydantic v2 — typed action / observation / state models
• —openenv-core — OpenEnv server and client base classes
• —OpenAI Python SDK — LLM fallback client (works with any OpenAI-compatible endpoint)
• —Hugging Face Inference Router — free access to open models (Qwen, Llama, Mistral, etc.)

No proprietary OCR, paid APIs, or closed models are required at any point.

Known limitations

• —Invoice text is synthetic OCR-like prose, not real scanned document output. A production version would pipe real OCR output through the same interface without changing the agent API.
• —The current public benchmark exposes one fixed scenario per difficulty level for maximum reproducibility. A richer version should include randomised invoice generation, adversarial formatting noise, and multi-page documents.
• —The heuristic agent uses regex patterns tuned to the three included scenarios; generalisation to unseen invoice layouts requires the LLM fallback path or a fine-tuned extraction model.