Pn101/taxbench-au

TaxBench-AU

A benchmark for testing whether AI agents can calculate Australian tax.

TaxBench-AU contains 156 Australian tax calculation questions, presented as multiple-choice (4-option) worked tax problems. The benchmark is designed to test whether an AI agent can read the facts, apply the right Australian tax rule for the relevant income year, do the calculation, and choose the correct answer.

The Kaggle mirror is published as Agent Tax Exam for Australian Tax.

Paper: PAPER.pdf Hugging Face dataset: https://huggingface.co/datasets/Pn101/taxbench-au Kaggle mirror: https://www.kaggle.com/datasets/poonam101test/australian-tax-mcq-benchmark

What is in the benchmark

The dataset contains 156 tax cases, grounded in current legislation and Australian Taxation Office (ATO) guidance.

The questions cover:

• —capital gains tax
• —rental property
• —depreciation
• —fringe benefits tax
• —individual income tax and offsets
• —superannuation
• —Division 7A and franking
• —GST
• —study and training loans
• —income tests
• —employment and termination payments

The questions range from simple one-step calculations to multi-step scenarios where the model has to keep track of several facts at once.

Each question includes four answer options, A to D. Correct answer positions are balanced across the dataset:

This prevents a model from benefiting from answer-position bias.

The benchmark is split into two groups:

All 156 public questions are synthetic. The original ATO worked examples are held out.

Files

All three CSV files join on the id column.

Quick start

from datasets import load_dataset

# Full benchmark with questions, answers, explanations, and metadata.
exam = load_dataset("Pn101/taxbench-au", "exam", split="test")
print(exam[0])

# Questions only for closed-book evaluation.
questions = load_dataset("Pn101/taxbench-au", "question", split="test")
print(questions[0]["question"])

# Answer key for grading.
answers = load_dataset("Pn101/taxbench-au", "answer", split="test")
print(answers[0]["answer_label"])

How to get your agent to take the test

1. 1.Feed your agent each row of data/question.csv. The question column is self-contained: it includes the scenario and A-D options.
2. 2.Ask the agent to return a letter (A, B, C, or D) or a numeric answer.
3. 3.Save the responses as a CSV with two columns: id,answer.
4. 4.Grade the submission:

python verifier.py grade my_answers.csv

The verifier prints overall accuracy plus a breakdown by topic.

Give your agent the calculator as a tool

Programmatic checking is the most reliable way to keep tax math accurate, so you can let the agent use the verifier as a calculator while it solves:

python verifier.py calc "(880000 - 615000) * 0.5"
python verifier.py verify --id Q001

The same python_verifier field is stored on every case in data/answer.csv and data/exam.csv, so every answer can be recomputed from a formula and named inputs.

For closed-book evaluation, block web access and tax-database retrieval. Otherwise, you are testing retrieval, not internalised tax knowledge. Retrieval can be a valid thing to test, but it should be measured separately.

What to measure

Overall accuracy is useful, but a good evaluation should also measure:

• —Per-topic accuracy: group results by tax area. A model may perform well on CGT but poorly on GST, FBT, or Division 7A.
• —Edge-case accuracy: score the 40 edge cases separately. This shows whether the model handles traps and boundaries, not just standard examples.
• —Robustness: paraphrase questions or make small wording changes. Accuracy should stay stable if the model understands the underlying rule.
• —Error patterns: use the labelled wrong options to identify common failure modes: missed thresholds, unavailable discounts, wrong income years, or skipped steps.

These measures make the benchmark more useful than a single score. They show where an agent is reliable and where it needs more work.

What this benchmark does not claim

This benchmark is a test tool, not tax or financial advice.

It is not endorsed by the ATO. All figures should be verified before anyone relies on them in real work.

The benchmark is tied to income years. Australian tax rates, thresholds, and rules can change. An answer that is correct for one income year may be wrong for another.

The questions are derived rather than verbatim. The dataset changes names and numbers, and the answer key cites the governing rule rather than relying on the original wording of the ATO example.

Licence

• —Benchmark items, documentation, and metadata: CC BY 4.0.
• —Verifier script and supporting code: MIT.
• —ATO source materials referenced by URL are © Australian Taxation Office for the Commonwealth of Australia and remain subject to the ATO's published copyright terms. See NOTICE.

This project is not affiliated with, endorsed by, or sponsored by the Australian Taxation Office, the Commonwealth of Australia, or the Tax Practitioners Board.

Citation

@misc{nair2026taxbenchau,
  title  = {TaxBench-AU: A benchmark for testing whether AI agents can calculate Australian tax},
  author = {Nair, Poonam},
  year   = {2026},
  note   = {Dataset and paper}
}