Solanabean/meta-ai

Forensic Financial Investigation — OpenEnv Environment

An RL environment where AI agents investigate synthetically generated transaction networks to uncover financial fraud. Built on the OpenEnv framework.

Hackathon / Phase 1: Deploy to a Hugging Face Space (Docker SDK), add tag `openenv`, then verify with python scripts/verify_hf_space.py <SPACE_URL> — details below.

Motivation

Financial institutions spend billions annually on Anti-Money Laundering (AML) compliance, and an estimated $2 trillion is laundered globally each year. This environment models the investigative workflow of a forensic financial analyst: receive an alert, explore a transaction network, identify suspicious patterns using graph analysis, and submit a report.

Every episode generates a fresh, randomized scenario with planted fraud patterns hidden among legitimate transactions. The agent must navigate the graph efficiently under a step budget, using tools like temporal burst detection, network centrality analysis, and cycle detection — mirroring real-world forensic techniques.

Architecture

┌──────────────────────────────────────────────────────┐
│  AGENT (LLM or policy)                               │
│  Observes case summary, entities, transactions       │
│  Takes actions: query / trace / flag / analyze /     │
│                 view / submit                        │
└────────────────────┬─────────────────────────────────┘
                     │  WebSocket (step / reset / state)
┌────────────────────▼─────────────────────────────────┐
│  FFI ENVIRONMENT (Docker / HF Space)                 │
│  ┌─────────────────────────────────────────────────┐ │
│  │  Scenario Generator                             │ │
│  │  Faker entities + Transaction graph + Fraud     │ │
│  ├─────────────────────────────────────────────────┤ │
│  │  Environment Logic (reset / step / state)       │ │
│  │  Investigation state, reward shaping, grading   │ │
│  ├─────────────────────────────────────────────────┤ │
│  │  NetworkX Graph Analysis                        │ │
│  │  Centrality, cycles, temporal, anomaly          │ │
│  ├─────────────────────────────────────────────────┤ │
│  │  Grader — F1-based + quality bonus, [0.0, 1.0] │ │
│  └─────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────┘

Tasks

Action Space

Every step, the agent submits one action:

Observation Space

After each step, the agent receives:

Observation lists are capped (25 entities, 60 transactions) to keep LLM token usage bounded. The network_stats field reports total counts and truncation status.

Reward Function

The reward provides signal throughout the trajectory, not just at episode end:

• —Correct flag (flag_suspicious on a fraudulent transaction): +0.1
• —False positive (flag_suspicious on a legitimate transaction): -0.05
• —Discovering a fraud-linked entity (query_entity): +0.02
• —Tracing a fraudulent transaction (trace_transaction): +0.03
• —Productive pattern analysis (analyze_pattern revealing relevant patterns): +0.01 to +0.02
• —Submitting report: F1 score as final reward + investigation quality bonus
• —Running out of steps: -0.1 penalty

Grading

Each task grader returns a deterministic score in [0.0, 1.0]:

score = F1(flagged, ground_truth) - false_positive_penalty + investigation_quality_bonus

Where:

• —false_positive_penalty = min(0.2, num_false_positives × 0.05)
• —investigation_quality_bonus = (fraud_entities_discovered / total_fraud_entities) × 0.05

The quality bonus rewards agents that thoroughly investigate the network, not just flag transactions blindly.

Key Design Decisions

• —No ground truth leakage: Flag messages are uniform regardless of whether the flag is correct, preventing agents from gaming the system.
• —Thread-safe scenario generation: A lock ensures deterministic ID generation even with concurrent requests.
• —Transaction metadata: Each transaction includes currency and channel (wire, ACH, SWIFT, crypto, cash_deposit), mirroring real financial data.
• —Graph-theoretic analysis: NetworkX powers the analyze_pattern action, providing real forensic investigation tools: betweenness centrality to find intermediaries, cycle detection for circular flows, and temporal clustering for structuring patterns.
• —Observation capping: Large observations are truncated with summary statistics to prevent token overflow in LLM agents.

Setup

Local Development

cd forensic-financial-investigation
pip install -r requirements.txt

# Run server
uvicorn server.app:app --host 0.0.0.0 --port 8000 --reload

# Test health
curl http://localhost:8000/health

# View tasks
curl http://localhost:8000/tasks

# Run heuristic baseline (no API key needed)
curl -X POST http://localhost:8000/baseline

Docker

docker build -t ffi-env .
docker run -d -p 8000:8000 ffi-env

Official submission inference (inference.py)

Hackathon validators run `inference.py` in the repo root. It uses only the OpenAI Python client and these required environment variables:

Optional: FFI_SERVER_URL (default http://localhost:8000), INFERENCE_BUDGET_S (default 1080 = 18 minutes wall time, under the 20 minute runtime cap).

Example (Groq — OpenAI-compatible):

export API_BASE_URL="https://api.groq.com/openai/v1"
export MODEL_NAME="llama-3.3-70b-versatile"
export API_KEY="gsk_..."          # or HF_TOKEN / GROQ_API_KEY for local only
export FFI_SERVER_URL="http://localhost:8000"   # or your HF Space URL

python inference.py

Example (OpenAI):

export API_BASE_URL="https://api.openai.com/v1"
export MODEL_NAME="gpt-4o-mini"
export API_KEY="sk-..."
export FFI_SERVER_URL="http://localhost:8000"
python inference.py

Stdout (for automated checks): only `[START]`, `[STEP]`, and `[END]` lines, each [KIND] + single JSON object (inference_log.emit_structured). The `[END]` line includes scores, average, model, and api_base_url. The same result object is duplicated on stderr as machine_result {...} for humans. All other logs use stderr.

Infra: Designed to finish within 20 minutes on 2 vCPU / 8 GB RAM (tight token limits per step, optional time budget).

Copy `.env.example` and fill in values locally (do not commit secrets).

Pre-submission checklist (all must pass)

Run the organizer’s pre-submission validation script (if provided) before uploading.

Baseline Scores

Heuristic baseline (/baseline endpoint, task-aware rules):

Grades are mapped to the open interval (0, 1) for validator compatibility. The clear progression (near-perfect → medium → lower) still demonstrates difficulty scaling. A capable LLM agent can improve on medium/hard tasks by using multi-step graph traversal, analyze_pattern for cycle detection, and pattern recognition.

API Endpoints

Trying /reset and /step in the browser

Paste-only URLs are GET requests. These routes are POST with a JSON body. Use `/docs` → Try it out, or:

curl -s -X POST "https://YOUR_SPACE.hf.space/reset" -H "Content-Type: application/json" -d "{\"task_id\":\"structuring\",\"seed\":42}"

Why GET /state looks empty

OpenEnv’s HTTP GET /state builds a new environment instance and returns its default state (episode_id null, step_count 0). It is not linked to a previous POST /reset. Multi-step runs use the WebSocket FFIEnvClient (/ws) — as in client.py and inference.py.

Hugging Face Space (Phase 1 submission)

1. 1.Create a Space at huggingface.co/new-space with SDK: Docker (matches this repo’s Dockerfile).
2. 2.Include `openenv` under `tags:` in the YAML block at the top of README.md (root of the repo). Hugging Face reads Space tags from that README card metadata — there is no separate tags control under Space Settings.
3. 3.Push this repository to the Space (git remote add / git push to the Space clone URL), or connect your GitHub repo if the UI allows.
4. 4.Wait for the Docker build in Logs (often 5–15 minutes).
5. 5.Verify:

   python scripts/verify_hf_space.py https://YOUR_USERNAME-YOUR_SPACE.hf.space

Expect: GET /health → 200, POST /reset → JSON with observation.

Troubleshooting: Build failures → check requirements.txt pins; 502 → cold start, retry; OOM → set WORKERS=1 in Space variables. Port must stay 8000 (app_port in frontmatter + Dockerfile EXPOSE). If opening the Space URL shows `{"detail":"Not Found"}` on /, use `/docs`, `/health`, or `/openapi.json` — the app is API-only unless you deploy a build that redirects / to /docs (current server.app does).