wealthschema/planning-benchmark

WealthSchema Planning Benchmark · now part of FiduciaryBench

New (2026-09): the `fiduciarybench` config carries the public split of FiduciaryBench's conduct suites — Reg BI suitability, RMD mechanics, and wash-sale mechanics — where every answer key is computed by code or derived from a quoted primary-source passage (SEC, FINRA, IRS/Treasury, U.S. Code) that exact-matches a locally held, versioned corpus. Per-item provenance pages, methodology, and the standing challenge policy: https://www.wealthschema.com/benchmark · Harness: pip install fiduciarybench. The planning configs below are FiduciaryBench's planning-calculations family.

Is your AI giving 2026 advice — or 2025 advice? Evaluation tasks for AI financial-advice systems, with answer keys built from primary-source-verified U.S. regulatory figure tables — including the enumerated wrong-but-plausible stale values (forbidden figures) with reason codes, so stale answers are countable, not anecdotal.

• —v2 (current): 50 tasks — the open split of the AI Eval Sets TY2026 corpus. Rule-grounding and threshold/cliff families, floor/standard difficulty. Each task carries answer_key.required_figures (value + unit + establishing source document, e.g. IRS Notice 2025-67) and answer_key.forbidden_figures (stale/superseded/derived values with reason codes: prior_year_value, superseded, derived_not_published, fabricated_forward_figure, …). Scoring is fully mechanical — no LLM judge.
• —v1 (superseded, kept for citation continuity): the original 14-task benchmark with expected/computation keys.

Headline metric: the Stale Figure Rate (SFR)

SFR = attempts asserting ≥1 forbidden figure as current ÷ figure-bearing attempts. In the pre-registered pilot of record (40 held-out tasks, 4 systems from 3 labs, k=3), 29% of figure-bearing attempts (86/300) cited a stale or fabricated regulatory figure; per-system SFR ranged 7%–63%, and errors repeated across attempts (pass^3 ≈ pass@1). Full protocol, substitution log, and per-system results: <https://www.wealthschema.com/resources/methodology/planning-benchmark-pilot-of-record> · live scoreboard: <https://www.wealthschema.com/benchmark>.

Using it

Blind evaluation: present prompt.system + prompt.user, k=3 attempts; score mechanically against answer_key (required figures present, forbidden figures absent as the operative value, typed expectations matched). The live API serves the same data with a blind mode: https://www.wealthschema.com/api/benchmark/v2?withhold_answers=true.

Report pass@1, pass^k, and SFR with the denominator, k, and vintage.

Splits and what's deliberately absent

This open split is 50 of a 471-task corpus. The adversarial categorical-flip family, all decision-recall tasks, and a held-out reserve (including all 40 pilot tasks) are never published — that is what keeps the maintainer-run scoreboard re-runnable and third-party comparisons meaningful. The commercial families are sold as one-time eval packs: <https://www.wealthschema.com/ai-eval-sets>.

License and training use

Free to use for evaluation, benchmarking, and research with attribution ("WealthSchema Planning Benchmark, wealthschema.com/benchmark"). `training_use_permitted: false` on every record — do not train, fine-tune, or include these tasks in training corpora; training on evaluation data destroys its value for everyone it is compared against. Figures are informational, cited to primary government documents, and are not tax, legal, or financial advice.

All scenarios are synthetic; no real persons, households, or accounts. Vintage: each answer key is correct for its stated tax year (vintage, as_of_date). Figures roll every January; a new vintage of this dataset ships each year.

Questions: support@capstera.com · <https://www.wealthschema.com/for-agents>