metaphilabs/credit-underwriting-preview

## Credit Underwriting Benchmark

A 1387-task RL environment for financial document understanding. Agents are provided with business documents (bank statements), and asked to assess the risk associated by using a context-rich classification taxonomy for each transaction . The agent's output is programmatically graded against the ground truth labelled by domain epxerts with over 10+ years of experience in the risk assessment field.

## Scale

• —1387 qualifying cases (Businesses, loan-origination)
• —5,607,517 total transactions
• —1-73 PDFs per case (median: 4, mean: 6.4)

## Task Definition

## Difficulty Distribution

### By PDF Count

### By Transaction count

### Financial Data Analyst (Extraction)

Extracts structured transaction data from PDF bank statements.

### Credit Analyst (Categorization)

Categorizes bank transactions by type, counterparty, and group.

## For RL Training

• —Reward signal: Extraction F1 as scalar reward (0.0 to 1.0), fully programmatic
• —Rollout: Agent receives PDFs → produces structured JSON → verifier computes F1 in seconds
• —Parallelizable: Each case is independent — runs in an isolated container
• —Difficulty calibration: Easy (84% F1) to hard (43% F1) — significant headroom
• —No human evaluation needed: reward signal is instant and deterministic
• —Decomposed rewards available: extraction F1 + categorization accuracy + counterparty accuracy as separate signals

## Dataset Schema

    {
      "id": "case-000",
      "case_id": "uuid-based-case-identifier",
      "pdf_gcs_uris": ["gs://.../_scrubbed.pdf"],
      "pdf_filenames": ["fileid_scrubbed.pdf"],
      "extraction_ground_truth": "{...}",
      "categorization_ground_truth": "{...}",
      "description": "3 PDFs - 792 txns",
      "case_type": "Business Case",
      "gt_status": "clean"
    }