caiotheodoro/lossbench-finance-v1

LossBench finance-v1

Severity-weighted expected-loss evaluation for agents that touch money. Three finance back-office domains, mechanical ground truth, and a contamination certificate. Models are ranked by what their mistakes cost, not by raw accuracy.

Overview

Tasks

Each task is an agentic back-office scenario with a deterministic seed, an executable verifier, and a gold outcome. Tasks are grouped by domain; every task carries a business severity band, a difficulty score, and a reference cost model. Models are ranked by expected loss under a cost model, not by raw accuracy alone.

License

This dataset is released under the cc-by-4.0 license.

Severity taxonomy

Every task is labeled with exactly one severity band:

• —LOW
• —MEDIUM
• —HIGH
• —CRITICAL

Severity bands describe the business impact of failure and are assigned at generation time, never inferred from model outputs.

Cost models

Severity costs are pluggable inputs, not constants of the dataset. Each cost model maps severity bands to business error costs K(σ); the dataset ships reference cost models (flat, reconciliation, principalrisk, reviewheavy) and adopters may register their own. Loss numbers therefore depend on the chosen cost model, and conclusions must be shown across a K range — sweeping the severity-cost scale as well as the pass@k window — so that rankings do not silently depend on one arbitrary operating point.

Results

Ranked by expected loss under the reconciliation cost model, lower is better. Seed 777, generated 2026-08-24T12:46:00+00:00, runner openai_compat. 60 tasks x 2 trials, single-shot (1 step, no corrective retry), via opencode zen go.

Run stopped after 4 of 6 models. leaderboard.json is reconstructed from the per-model cards, so ECE, parse rate and error rate are not included: those live in the run summary, which was never written.

• —Expected loss charges each unreviewed error the severity cost K of the task it got wrong, so a HIGH miss outweighs a pile of LOW ones. It is not accuracy.
• —Token cost is deliberately omitted: the repo prices runs from a placeholder rate table, not from what the gateway actually billed.

qwen3.7-plus answers more of the suite correctly than qwen3.6-plus (0.933 vs 0.917) and still costs 3.7x as much (13.4 vs 3.6), because its mistakes land on higher-severity tasks. Accuracy and expected loss rank these two in opposite orders, which is the reason this benchmark exists.

Honest limits

• —Tasks are generated, not observed. They are representative of back-office workloads by construction and citation, not by sampling.
• —Severity costs are contested inputs, not constants. Every conclusion must be shown across a range of cost models.
• —Ground truth is mechanical: the verifier recomputes the answer from initial_state alone and a task is only kept when it agrees, so verifier agreement is enforced at generation rather than measured.
• —The decision rules are stated in each prompt. This benchmark measures whether a model applies a published policy to structured data, not whether it can guess an unpublished convention.
• —Results are a single run on one seed against one gateway. Small n; treat the absolute numbers as indicative, not as a leaderboard.

Contamination policy

Train is seed 101, eval is seed 777. A task signature is a SHA-256 over every field except id, seed and signature, so a renumbered copy of the same content still collides. Measured overlap between the two splits is 0.0. Publishing is refused if it is anything else. Signatures are excluded from the published rows so the eval set cannot be fingerprinted from this dataset alone.

Reproducibility

Every row is a pure function of (generator version, domain, seed, index). Regenerate with generate_suite(domain, seed, n_tasks) from https://github.com/caiotheodoro/lossbench. The same seed yields byte-identical tasks, which the repo enforces as a CI gate. Because identity is code-version dependent, cite a revision rather than main.

Contact

https://github.com/caiotheodoro/lossbench/issues