wealthschema/planning-benchmark

1---2license: other3license_name: wealthschema-benchmark-license4license_link: https://www.wealthschema.com/license5task_categories:6- question-answering7language:8- en9tags:10- finance11- financial-planning12- benchmark13- evaluation14- tax15- llm-evaluation16- stale-figure-rate17pretty_name: WealthSchema Planning Benchmark18size_categories:19- n<1K20configs:21- config_name: fiduciarybench22  data_files:23  - split: public24    path: data/fiduciarybench-public.jsonl25- config_name: v226  data_files:27  - split: open28    path: data/planning-benchmark-v2.jsonl29- config_name: v130  data_files:31  - split: open32    path: data/planning-benchmark-v1.jsonl33---34 35# WealthSchema Planning Benchmark · now part of FiduciaryBench36 37**New (2026-09): the `fiduciarybench` config** carries the public split of38FiduciaryBench's conduct suites — Reg BI suitability, RMD mechanics, and39wash-sale mechanics — where every answer key is computed by code or derived40from a quoted primary-source passage (SEC, FINRA, IRS/Treasury, U.S. Code)41that exact-matches a locally held, versioned corpus. Per-item provenance42pages, methodology, and the standing challenge policy:43https://www.wealthschema.com/benchmark · Harness: `pip install fiduciarybench`.44The planning configs below are FiduciaryBench's planning-calculations family.45 46**Is your AI giving 2026 advice — or 2025 advice?** Evaluation tasks for AI47financial-advice systems, with answer keys built from primary-source-verified48U.S. regulatory figure tables — including the enumerated wrong-but-plausible49stale values (*forbidden figures*) with reason codes, so stale answers are50countable, not anecdotal.51 52- **v2 (current):** 50 tasks — the open split of the AI Eval Sets TY202653  corpus. Rule-grounding and threshold/cliff families, floor/standard54  difficulty. Each task carries `answer_key.required_figures` (value + unit +55  establishing source document, e.g. IRS Notice 2025-67) and56  `answer_key.forbidden_figures` (stale/superseded/derived values with reason57  codes: `prior_year_value`, `superseded`, `derived_not_published`,58  `fabricated_forward_figure`, …). Scoring is fully mechanical — no LLM judge.59- **v1 (superseded, kept for citation continuity):** the original 14-task60  benchmark with `expected`/`computation` keys.61 62## Headline metric: the Stale Figure Rate (SFR)63 64SFR = attempts asserting ≥1 forbidden figure as current ÷ figure-bearing65attempts. In the pre-registered pilot of record (40 held-out tasks, 4 systems66from 3 labs, k=3), **29% of figure-bearing attempts (86/300) cited a stale or67fabricated regulatory figure**; per-system SFR ranged 7%–63%, and errors68repeated across attempts (pass^3 ≈ pass@1). Full protocol, substitution log,69and per-system results: <https://www.wealthschema.com/resources/methodology/planning-benchmark-pilot-of-record>70· live scoreboard: <https://www.wealthschema.com/benchmark>.71 72## Using it73 74Blind evaluation: present `prompt.system` + `prompt.user`, k=3 attempts;75score mechanically against `answer_key` (required figures present, forbidden76figures absent as the operative value, typed expectations matched). The live77API serves the same data with a blind mode:78`https://www.wealthschema.com/api/benchmark/v2?withhold_answers=true`.79 80Report pass@1, pass^k, and SFR **with the denominator, k, and vintage**.81 82## Splits and what's deliberately absent83 84This open split is 50 of a 471-task corpus. The adversarial85categorical-flip family, all decision-recall tasks, and a held-out reserve86(including all 40 pilot tasks) are never published — that is what keeps the87maintainer-run scoreboard re-runnable and third-party comparisons meaningful.88The commercial families are sold as one-time eval packs:89<https://www.wealthschema.com/ai-eval-sets>.90 91## License and training use92 93Free to use for evaluation, benchmarking, and research with attribution94("WealthSchema Planning Benchmark, wealthschema.com/benchmark").95**`training_use_permitted: false` on every record** — do not train,96fine-tune, or include these tasks in training corpora; training on97evaluation data destroys its value for everyone it is compared against.98Figures are informational, cited to primary government documents, and are99not tax, legal, or financial advice.100 101All scenarios are synthetic; no real persons, households, or accounts.102Vintage: each answer key is correct for its stated tax year (`vintage`,103`as_of_date`). Figures roll every January; a new vintage of this dataset104ships each year.105 106Questions: support@wealthschema.com · <https://www.wealthschema.com/for-agents>107