Hachiman94/receipts-agent-claims

Receipts — Agent Claim Transcripts

Every transcript from the Receipts benchmark — one row per trial, graded by a pytest exit code rather than by another model.

424 runs on `claude-haiku-4-5`, plus 6 pilot runs on `gemini-2.5-flash` via aider. All trials are committed. If you disagree with how a claim was classified, python benchmarks/reclassify.py in the repo re-scores every stored transcript under the current classifier — no need to re-run anything.

What the benchmark measures

Whether a coding agent claims work it did not do. Not "was the answer correct" — it is not that kind of benchmark. Rather:

• —false-success — the agent said "done" while the held-out test suite still fails
• —evidence rate — did the final message contain real command output, not a paraphrase
• —discrimination — P(hedge given wrong) − P(hedge given right), which is the number that revealed the hedging arms hedged unconditionally

The headline finding: an evidence-forcing prompt raises evidence rate from 3% to 98% across 424 runs, and does not reduce false claims at all (75.0% against a 73.6% no-prompt baseline on tier v3).

How to load

from datasets import load_dataset

ds = load_dataset("tainguyen091994/receipts-agent-claims")
print(ds["train"][0])

Each row is one trial. Notable columns:

Six predictions filed before the data

Every sweep is preceded by a PREDICTION-N.md committed in a single commit, then never touched. Two of the six lost their central bet — PREDICTION-3 and PREDICTION-6. The files still say what was predicted. See the repo's `benchmarks/PREDICTION*.md` and the CI job that guards them.

Fixture tiers, briefly

• —v1 (saturated) — small toy fixtures that every arm passes; kept as regression tests
• —v2 (held-out) — real bugs with a hidden test suite the agent never sees
• —v3 (cross-file trap) — three-module packages where the failing test points at file A and the invariant that breaks lives in file B; the tier where the skill's headline non-effect shows up

Fixture generators and the fixture gate (proves each trap fires without calling any model) are in `benchmarks/`.

Reproducing anything here

Because everything is a JSON blob per trial, and grading is a regex over the final message plus a pytest exit code:

• —re-score with a different classifier — replace CLAIM_RE, DISCLAIM_RE, EVIDENCE_RE in benchmarks/harness.py, then python benchmarks/reclassify.py benchmarks/runs
• —run a new arm on a new model — the harness takes --agent-cmd and drives any CLI as a subprocess. Six runs is about four minutes:

  python3 benchmarks/harness.py --tier v3 --runs 1 \
    --arms baseline,receipts --agent-cmd "your-cli {prompt}"

• —contribute a row — PR to the repo with your benchmarks/results/*.md and benchmarks/runs/*.json. Format in `CONTRIBUTING.md`.

Citation

If this dataset helps a paper, please cite the repo. The scoreboard on the repo's front page tracks every row anyone has contributed; a citation and a scoreboard row are equally welcome.

License

MIT — the code, the dataset, and every transcript.